Hi, I want to know what is the best way to keep the databases I use in different projects? I use a lot of CSVs that I need to prepare every time I’m working with them (I just copy paste the code from other projects) but would like to make some module that I can import and it have all the processes of the databases for example for this database I usually do columns = [(configuration of, my columns)], names = [names], dates = [list of columns dates], dtypes ={column: type},

then database_1 = pd.read_fwf(**kwargs), database_2 = pd.read_fwf(**kwargs), database_3 = pd.read_fwf(**kwargs)…

Then database = pd.concat([database_1…])

But I would like to have a module that I could import and have all my databases and configuration of ETL in it so I could just do something like ‘database = my_module.dabase’ to import the database, without all that process everytime.

Thanks for any help.

  • gedhrel@lemmy.world
    0·
    5 months ago

    There’s not much here to go on. Are you asking how to write a module that you can import?

    Are these the same set of DB files every time? Are the columns and other configurations the same? Are you writing new python code every month?

    Are you using some ETL process to spit out a bunch of files that you’d like to have imported and available easily? Are the formats the same but the filenames differ?

    I think it’s the first thing you’re after. There are a bunch of tutorials knocking around about this, eg, https://www.digitalocean.com/community/tutorials/how-to-write-modules-in-python-3

    You might also be asking: if I write a module, how do I make it available for all my new python projects to use? You could just copy your whatever-my-module-is-called.py file around to your new projects (this might be simplest) but if you’re also expecting to be updating it and would like all of your projects to use the updated code, there are alternatives. One is to add the directory containing it to your PYTHONPATH. Another is to install it (in edit mode) in your python environment.

    [I get the impression you’re a data person rather than a programmer - perhaps you have a colleague who’s more of the latter you can tap up for this? It doesn’t have to be difficult, but there’s typically a little bit of ceremony involved in setting up a shared module however you choose to do it.]

    • driving_crooner@lemmy.eco.brOP
      0·
      5 months ago

      Are you asking how to write a module that you can import?

      Yes, kinda.

      Are these the same set of DB files every time? Are the columns and other configurations the same? Are you writing new python code every month?

      They get updated by the accounting team each month. Some of them are csv, other come from an access database file, other from the sql server.

      Some of the code need to be run each month with the updated databases, but there’s a lot of ad hoc statistical studies that my boss ask for that use the same databases.

      Are you using some ETL process to spit out a bunch of files that you’d like to have imported and available easily? Are the formats the same but the filenames differ?

      I guess yes. And not, the accountants keep the same filenames but change the directory lmao.

      I think it’s the first thing you’re after. There are a bunch of tutorials knocking around about this, eg,

      Thanks, im checking it out.

      how do I make it available for all my new python projects to use?

      import sys sys.path.append('my\\modules\\directory) import my_module

      I get the impression you’re a data person rather than a programmer -perhaps you have a colleague who’s more of the latter you can tap up for this?

      You’re right, I’m an actuarie. I wanted to do computer science instead of actuarial sciences, but I tough that it would be better getting an actuarial degree and then doing a masters on CS (still in planning, maybe 2026). I’m the only guy on the company who uses python and people here thinks I’m a genius because I have automated some boring things from excel.