user-pic

Upgrading from 4.1 -> 4.21 with huge database runs out of memory

Vote 0 Votes

Hi,

I'm trying to upgrade movable type to 4.21 and it always runs out of memory when doing task "Assigning all permissions to blog administrator...". This was on a machine with 8Gb of RAM and a 6Gb swap.

This task applies all the new permissions to Blog Administrators or something like that. It goes through the mt_permission table. We have more than 3000000 rows there.

I was thinking as a workaround to just comment that part from the Upgrade.pm file. And afterwards just do some changes in the Blog Administrator role for the permissions to get calculated again.

BTW. I don't see any need for mtpermission table. It's just a join of mtassociation and a few other tables. It should be done with a view or something. Correct me if I'm wrong

Reported on Movable Type 4.2

3 Replies

| Add a Reply
  • I forgot the question :)

    I'm testing my workaround and what might be the caveats in doing it.

  • The problem is that mt_permission controls access controls. It's really not a table that you can just forget about.

    Look up the Unix command "ulimit." I'm not a sysadmin, so you'll probably need to experiment with this, but what I would try would be to put (I believe this is the syntax) "ulimit -u unlimited" at the top of the script that starts Apache, and if that doesn't work, try putting "#!ulimit -u unlimited" at the top of mt.cgi.

    ulimit allows you to override memory limits that are put into place as security measures on Linux. This way, theoretically, your mt.cgi script could take several gigs of memory, if needed, to complete the operation. Chances are, right now, it's only getting a few dozen or a few hundred megs of RAM. I know from experience that no matter how optimized your code is, or what language you're running into, having 3,000,000 value objects in memory that represent database records is going to take A LOT of RAM, and probably a lot more than ulimit allows by default.

  • One more thing. If and when you get this working, absolutely don't forget to take ulimit -u unlimited out of those scripts because that command will prevent your server from defending itself from attacks that cause Apache to allocate more and more memory to itself.

Add a Reply

Forum Groups