Switch model between DEV and PROD

#1

Hi,
I have 2 environments:

  • DEV : test environment
  • PROD : production

If I have to train a machine learning model on PROD, there is a way to train it on Jupyter on DEV and upsert it also in PROD environment?

Thank you

0 Likes

#2

Assuming your model is small enough, you can get your pipeline (if necessary make sure to include everything), serialize it to JSON and copy paste it in your PROD console.

Something like this:

  1. Go to DEV console
  2. MLSerialPipeline.get("pipelineId").toJsonString()
  3. Copy result without the quotes
  4. Go to PROD console
  5. var modelObj = ... // Paste here (without the quotes)
  6. MLSerialPipeline.make(modelObj).upsert()

If you model is big, let’s discuss to see what you are really trying to do.

0 Likes

#3

There is some debate about this in C3, but i believe this is very bad practice and likely to lead to errors. You should not directly change the running code in production without going through a formal validation process (CI/CD). This could cause errors and unexpected behavior.

I know that this is a data science thing to push your experiment to production, but i think you would have a different opinion the first time you bring down a mission critical application (which i have DEFINITELY done).

Anyway, the choice is yours, and you have heard my warning. Good Luck!

0 Likes

#4

I agree with Riley in principle (there is some context here not explained in the question).

Cleanest way would be to test end to end training + model use for inference.

Now, if you are not doing training in the platform, or if training is very hard to test, one thing you could do that would still allow you to test your model inference and give you some guarantees is to:

  1. repeat steps 1 to 3 (export json)
  2. paste in some seed data file in your application
  3. run your prediction test using that model
  4. provision/promote your application to PROD after your test is green
1 Like