Hello everyone! Again, I have a problem with my gigantic LP (3 million variables) that I want CPLEX to solve. So the linear program basically describes objects of in images moving over time. It works perfectly fine when I give a manual upper bound for the time (e.g. I only parse 20 time steps). But when I use the full data (100 time steps), it gets weird, namely: The dual objective value changes each iteration in a reasonable way, e.g. it goes up to 263 (I am expecting a number between 250 and 300). But after some thousand iterations, CPLEX removes a perturbation and boom, the objective goes down to millions. I read that a resetting of the Markovitz Threshold (see the log) is an indicator of numerical problems. Could that be the case? I have tried to tell CPLEX to not change the dual norms, but now it is solving since 30 minutes and the dual objective is still only around 175. I really have no idea what is going wrong, especially since the program works fine for a subset of the data. Has anyone experienced this before? I would appreciate your tips very much! Let me know if you need more information about something in particular. Thank you very much and have a nice day! Here is part of the log:

Disclaimer : I am no cplex expert and you will probably get a better answer at cplex support forum. When cplex removes perturbation your basis is most likely refactored and judged instabil, which is why they also choose to raise the markowitz tolerance to increase stability on internal solves in the simplex algorithm. The reason why you see huge change in objective can be either :
Since Cplex is a state of art optimizer, I doubt it's second bullit point but one is much more likely. These things can occur, I would advise you to investigate your data to examine for badly scaled data. If this happens a lot, you could (only on these types of models) raise the default level of the markowitz tolerance, but generally I am against too much fiddling with standard parameters. You could also set a maximum iteration count which equals just before it goes wrong, when it has stopped print out the condition number of the basis matrix stored inside the solver. If it's high, then we are definitely in case one. One thing I think is a bit strange, is they start to recount iterations, I don't remember having seen a log with this before i.e in same solve. BTW : If you in anyway can share the data, I would be interested to take a look at this LP model, in such case please contact me, see profile for details.  answered 12 Jul '12, 09:38 Bo Jensen ♦ 
Care of Alexandra Newman and Ed Klotz via an email discussion, Try: 1) Playing with the numerical emphasis parameter. 2) Reading the problem into interactive cplex and displaying the problem stats to see if the problem is poorly scaled. If there are more than 10 orders of magnitude separating the largest and smallest numerical values in A, b, and/or c, then the data need to be rescaled. answered 14 Jul '12, 16:35 Carleton 
I don't have my CPLEX docs handy (and, oddly, the coffee shop I'm sitting in doesn't seem to have a copy), but you should be able either to track basis condition numbers (kappa values) or stop the optimization when a negative one zillion dual objective occurs and grab the kappa for the last basis. That will let you judge whether it's a numerical instability problem (which I think highly likely). Plus, I second what Bo said. answered 13 Jul '12, 18:01 Paul Rubin ♦♦ Cplex manual  The starbucks version for the "salesman" on the move...
(13 Jul '12, 18:05)
Bo Jensen ♦

I warmly thank Bo, Carleton, and Paul for their useful answers. As said, you could (should?) ask such product specific question on our support forum: http://www.ibm.com/developerworks/forums/forum.jspa?forumID=2059 answered 16 Jul '12, 06:23 jfpuget 
Hello everyone!
Thank you very much for your suggestions. I only work one day a week thus I couldn't try them out yet, but I will tomorrow. I will let you know of the results.
Bo: Unfortunately I cannot share the data, I am not allowed to.