We've also been able to apply automated checkpoint age monitoring to other write-heavy operations, proving that investing in the deep understanding of our tools certainly pays off (and deleting one million things per hour is unbelievably satisfying). And though it's taken several iterations to get here, the amount of technical debt it's freed up is well worth it. Going from too slow to too fast, we finally found the goldilocks of transfer deleting speeds. The code has been up and running unattended without a single problem yet (knock on wood). With this final addition, the problem appeared to be solved. We can also verify our checkpoint age after each of those queries to make sure we're not overloading the database: until num_deleted MAX_ALLOWED_CHECKPOINT_AGE So instead of having a single large query of an unknown length, we can chop it up into an unknown (but finite) number of transactions of a known length. Luckily MySQL supports LIMIT clauses on DELETE queries. We found this out the hard way when a single transfer (uploaded by a certain mononymous musician) was downloaded over 19 million times. Deleting all those rows in one go could still exceed the maximum checkpoint age with a single query, rendering our protective check pretty useless. While most transfers are only downloaded a handful of times, some are downloaded in the millions. Some_transfers_to_delete = Transfer.where(delete_at ((2/3) * MAX_ALLOWED_CHECKPOINT_AGE)Īfter this, the checkpoint age no longer reached unsafe levels - success. Naturally, our first attempt was something like this: until finished() do All our database tables are ActiveRecord models and they already had the relevant relations defined between them. Knowing which transfers we needed to delete was straightforward since the tables have an indexed delete_at column. So, as WeTransfer continued to expand and develop, we decided to get to the bottom of the issue once and for all. Plus, these tables grow to multiple terabytes in size making them problematic to store.ĭeleting so much in such a short space of time and at such a regular cadence was taking its toll.įor a long time, we would actually take the site offline for a couple of hours throughout the year and remove the rows manually, since trying to do it while the site was up would result in mysterious downtimes. Not deleting the metadata would be a violation not only of people’s trust and expectations but also of regulations like the GDPR. So when it comes to deleting 20 per second, that’s a serious amount of rows to take out in one go.Īnd we’ve got to stay on top of it. Often transfers can contain hundreds of files and potentially millions of downloads. The database tables have a simple structure with several child tables (containing things like download details and file metadata) belonging to a single parent table called transfers. Why is uploading in WeTransfer so slow When this happens, something on your end might prevent the uploaded from reading your transfer correctly. We keep track of uploaded transfers, their senders and their intended recipients in a centralized MySQL database. Sounds pretty straightforward but, with an average of 20 transfers uploaded every second, that’s 20 transfers we need to delete per second just to keep up.Īnd sadly, no, we can’t just hit ‘delete all’. Most of these transfers expire after a week which means they need to be deleted. But what about when the files reach B - what happens after that? Transfer KonmariĪbout 55 million transfers are created every single month on. Simply, safely, and (dare we say so ourselves?) beautifully. I just did a test of uploading 700 raw photos and it took the desktop app a little over 3 minutes while the browser still has over 45 minutes to go.Wander Hillen Backend Engineer November 18, 2019įor ten years people have known WeTransfer as the go-to service for getting their stuff from A to B. I use a few other services that require browser uploading (WeTransfer, Vimeo, Youtube, Pixieset, GoogleDrive) and Dropbox is the only one that is slow. Is there a way to use the desktop app to share files with someone else's dropbox link? If not, is there are way to make the browser upload faster? Also this connection is frequently dropped and the uploader shows as if all the files are uploaded, but when checking we see that some files are missing, sometimes a lot, sometimes a few. I've tried using different browsers (Chrome, Firefox, Edge and Opera) or different machines (Win11, Mac and a linux distro) and it will not change how fast it goes. These apparently can only be done through a browser, at least I haven't found a way to use the app yet, which takes absolutely ages. My main issue: Some of my peers also use dropbox and will send me upload requests and vice/versa. I currently use Dropbox for sharing files and upload my files mostly using the desktop app, which uses my full internet connection of 1Gbps, usually easy and no errors.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |