Skip to content

May 18, 2015

I wrote about the Mysql storage in EBS volume problem last week. That problem still persists, and will work on that to fix it totally.

But good news is, meanwhile found the reason for mobile process getting killed. The Linux OS was killing these processes because of them consuming a lot of memory. dmesg commands

Output of ‘dmesg | grep mobile | grep -i kill’ :

[492305.469363] Killed process 29371 (mobile) total-vm:3974224kB, anon-rss:3775748kB, file-rss:0kB
[493341.992196] Out of memory: Kill process 15303 (mobile) score 209 or sacrifice child
[493341.992203] Killed process 15303 (mobile) total-vm:1790616kB, anon-rss:1592852kB, file-rss:0kB

Its going so high in memory usage (3.9 GB!) , because of sessions stored in the memory. So I am trying to move the session storage to a local file. Which will be on the instance-attached SSD storage. Obviously not on EBS! (As that’s a battle I am already fighting with MySql storage).

Initial results: By evening, I have deployed the mobile service on all the servers, to work with local file based session storage, and the initial results seem to be good. Some specific improvements on top of this change:

1) Optimize the number of reads. Read only once. In memory version was slightly lax (as sessions were in a hash map — constant time read)

2) Store asynchronously. Now since we write to a file, and since array reads can happen in multiple threads, we can call ‘go writeSessionToFile(… )’

3) Now since sessions are stored in file. The fail in stickiness at server level (nginx was seen to send 8-10% of requests to the wrong mobile service) won’t effect as session can still be fetched.

Fingers crossed. Will come to know the actual improvement of this change in a day’s time. No (or very less) crashes) is what I am looking for.

Advertisements

From → Uncategorized

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: