Shipping costs in the API

Markuzzz

The API currently does not allow to retrieve shipping cost information on stores.
The API does list all the lots that are available for a specific item (../catalog/availability), but this information is somewhat meaningless without knowing the associated shipping costs.

For context, I am pretty far along in building a multi-store part buying tool (similar to BrickLinks easy-buy tool) that is able to automatically search for a cheap distribution of parts over stores. An important factor in this algorithm is the shipping cost of each store.

Suggestion #1: For each lot returned by the ../catalog/availability API endpoint add a field "shipping_cost" with the cost of shipping just a single item from that lot. This seems like a minor modification. For people using the API to find good deals on sets this may already be sufficient. For my algorithm I would prefer to have more information, I also need to know what it costs to buy multiple parts, but having a baseline like this would already be nice.

Suggestion #2: Create a new API endpoint from which all the information on a store can be retrieved, including detailed information on the shipping prices. As argument it would take a country code indicating the shipping destination. As a response it should then give a mapping from store_ids to the information on the store. The information could contain both the shipping tiers and the same store information returned by the ../catalog/availability API endpoint. As an example of what the JSON could look like see the attached example_json.txt file.

Suggestion #3: if the second suggestion is indeed followed then it would make sense to optionally remove the store information from the ../catalog/availability API endpoint. If I retrieve the lots for a couple of hundred different parts I get the same store information many times over, which is rather inefficient. An optional argument ‘store_info’ could be added that defaults to true (to not break existing code). If the argument is set to false only the store_id would be included.

Markuzzz

For those interested, the problem of finding the cheapest stores to buy parts using an algorithm is very interesting. In the scientific literature it is known as the Internet Shopping Optimisation Problem (ISOP). It is known to be NP-hard, which means that any algorithm that is guaranteed to find the optimal solution will take exponential time in terms of the input (the number of parts and number of stores). The search space is mind-bogglingly huge. If you have just 200 parts and 100 stores (that all sell all parts) then there are 100 to the power of 200 distributions of parts over the stores. That is (way) more than the number of atoms in the universe.

Finding an optimal solution is then completely infeasible, you need to be happy with finding a 'good' solution. Over the past year I have experimented with lots of algorithms and have settled on one that seems to work pretty well (on random synthetic data). I have already integrated the real data from BrickOwl on which store is selling which lot, the only missing puzzle piece is the shipping information.

I am hoping to launch the tool with a web interface in the summer. I think it could be very beneficial to the BrickOwl community. As a side note, I am a computer science PhD student and this has been a very entertaining hobby project.

Musje7

I think this is a great idea; very useful.

Colourfan

You are building a optimization tool for buying parts?!? That would be só awesome to have, I'm rooting for your project!
Cheaper shipping, cheaper projects,... Then I can do more projects! Ha!
I heard there used to be a tool long ago for desktop, but that doesn't work anymore. Having one on the web would be totally awesome.

Markuzzz

@Colourfan , yeah there used to be a tool called Brickficiency but it is no longer maintained and doesn't work anymore.

Markuzzz

I just tested my algorithm by comparing the results with BrickLinks store searching tool. The results are shockingly amazing.

For the test, I took the Fondor Haulcraft MOC: https://rebrickable.com/mocs/MOC-137520/Eventus_Engineering_System/fondor-haulcraft-ucs/#parts
I only included stores within the EU, with shipping to The Netherlands.

These are the game changing results:
BrickLink: E 744,49
BrickOwl (picking biggest stores): E 719,88
My tool, with BrickOwl data: E 512,70 !!! That's E 200 less, 30% cheaper!!

For those interested, here are more detailed results:
Bricklink comes up with a selection costing 744,49 euros (including shipping and handling).
Using the BrickOwl store selection tool and picking the store with the most needed lots each time gets me a selection costing 709,88 euros and missing the shipping costs of one store (quote only). Lets say the shipping costs are 10 euros then the final total 719,88 euros.
Using my tool to determine which stores to buy from and then selecting those stores in the BrickOwl store selection tool get a solution of 492.70 euros! Two stores are missing shipping so lets add 20 euros. The grand total then is 512,70 euros. This saves 200 euros for a single MOC. And then I can't even optimise on shipping yet!

This is a game changer!

You can check the results by yourself, here are the stores my tool selected:
FamilyBricks
MARYLAND
FloathBricks
Buy cheap in bulk!
MarcoDea1907 Brick House
Magicshop
Klemmnix
Sunflower_Bricks_Shop
BricksByTheRiver
Bricks Magic
JoanyBrick

P6tu

Wait, wait - this is something that I'm looking for due to huge amount of needed parts. Where can I test it or how can I have a test run with it? There is a public wishlist for my project:
https://www.brickowl.com/wishlist/view/P6tu/the-base
The list is still a subject of change and I am not ready to start ordering the needed spares (at least not yet), but the majority of the project is ready... Now it would be amazing to find a way to get the needed spares with the most optimal way. Unfortunately Brick Owl Magic isn't helpful with such a big list of things, but your idea and your project sounds quite what I will be needing in the future.

PS! Awesome project and somewhat sounds like something of a thesis work for a PhD degree .

Markuzzz

Hi, thanks for showing interest! This project actually has very little to do with my PhD project. Though you could say that doing the PhD has given me the skills to find a good algorithm.

Right now I can only import rebrickable CSVs into my tool as wishlist. I will soon add the ability to import BrickOwl wishlists. I plan on launching a beta version of the tool somewhere this month so you can try it out then. I will let you know when you can try it out as one of the first users!

Calibrick

@Markuzzz, how do you get around the need to know the prospective buyer's zip or other postal code format in the systems? Very few stores charge flat rate, e.g., in the US, my shipping costs is based on my start point zip code vice the buyer's zip code, which determines the postal zone that applies.

Also, I'm assuming it recalculates the shipping rates when parts or added or removed from the list? As that may change the shipment size, which could change the shipping rate...

I'm super-intrigued with this! All us folks that part out sets or are MOCers, well, this is what we desperately want to see. :-) Good for you!!!

Markuzzz

@Calibrick , ideally I would want to know exactly how BrickOwl computes the shipping costs based on the shipping methods of the seller, the location of the buyer (country + zip code) and the list of items to buy at the store (weight, volume, etcetera).

My algorithm could then optimise on that as precisely as possible. It could for example try to stay just below a weight treshold for the shipping option. To be able to do that I need the info from BrickOwl; that is why I created this suggestion.

In the end, the more information I have, the better I can optimise. Once the site is live (in a couple of weeks I hope) I will post a message in the forum again so you can check it out.

Markuzzz

Hello @Calibrick , @P6tu , @Musje7 , @Colourfan , and all others reading this thread: I am pleased that a first version of my tool has been released. You can check it out at brickonomics.net !

Brickslan101

How do you get the link to the Rebrickable CSV, i can only export the CSV and it downloads it. I don't get a link.

Edit : Figured it out.

Markuzzz

@Brickslan101 , you should right click the "Rebrickable CSV" option. You will then have the option to copy the link. On mobile this may be more tricky. On my phone I can long press the "Rebrickable CSV" option and it will then give me the option to copy.

P6tu

Hi,

I will check it out . Sorry for not replying to your messages. Daily life has overwhelmed me a bit and I have not had any time to deal with my MOC-s for the past few months ... I will check it out, once I get my my responsibilities on track.

With regards,