BrickSync, inventory synchronization software

All right, here's finally a first beta release of BrickSync! This is a small piece of software meant to run on a seller's computer, keeping inventories synchronized between BrickOwl and BrickLink, with many useful features beyond that. Don't be scared by command-line software, it shouldn't be too complicated to use (I'm a Linux zealot who hasn't touched Windows in 10 years, but I made a real effort :p).

First step: Unpack the package somewhere accessible, there's no installer.

Second step: Modify the file data/bricksync.conf.txt and enter API keys for both BrickOwl and BrickLink.

Third step: Run bricksync, it will read and compare your BrickLink and BrickOwl inventory. It will ask your permission to apply all changes required to make the BO inventory match the BL one.

Fourth step: You are done! When new orders arrive, they will be stored on disk as BrickStore/BrickStock files, and subtracted from your inventory. You should now avoid editing your inventory on BO or BL, everything goes through BrickSync.

Some commands:
status ; Hey BrickSync, how are you feeling today?
help ; General help, not much there yet
check ; Force a "check" of both BO and BL for new orders
sync ; Force a "deep sync" of both BO and BL, making sure they exactly match the local tracked inventory
add ; Add a whole BrickStore/BrickStock .bsx file to the inventory. Quantities are merged, but other fields are *not* modified for existing lots.
sub ; Subtract a whole BrickStore/Brickstock .bsx file from the inventory. Also useful to "return to inventory" a cancelled order.
loadprices ; Read a .bsx file and update all prices for existing lots. Other lots are ignored.
loadnotes ; Read a .bsx file and update all comments/remarks for existing lots. Other lots are ignored.
sort ; Update a .bsx file with the comments/remarks/price filled in from the tracked inventory for all matching items. Items are also sorted by BL color then by BL name, making it easy to combine new parts with your physical inventory using BrickStore/BrickStock.
item ; Print information on a specific item (can reference it by BLID, BOID, BlLotID and BoLotID).
find ; Run a search through all items for specified keywords, it prints items matching all arguments (i.e. find 3001 "uish Gray").
setquantity ; Set quantity of a specific item, relative or absolute.
setprice ; Set price for an item.
setcomments ; Set comments for an item
setremarks ; Set remarks for an item

There are a bunch of commands left undocumented until finished and polished. It's potentially useful stuff, such as asking BrickSync to evaluate how parting out a specific set would improve the inventory (what parts are missing from the inventory, their value, demand/offer ratio, etc.).

Note that BrickLink has a limit of 5000 API calls per day, so be careful with the loadprices/loadnotes commands, as each lot to be updated is one API call. I need to add some commands to output BL XML files to upload manually as an alternative...

Other BrickSync features:
- All orders coming from both services are saved as BrickStore/BrickStock .bsx files.
- Constant inventory backups are saved as .bsx files along with verbose log files.
- If any inventory update to a service fails with ambiguous results (i.e. we never had a reply to a query sent), the code automatically peforms a deep synchronization to detect and resolve any issue.
- Data journalling to recover smoothly from any interruption or power cut (absolute robustness bypassing the hard drive's internal caches only guaranteed on sane Unix platforms, OSX not included).
- Local database for BOID<->BLID translation continuously built on the fly as it sees items passing by (therefore BricLink can't claim the software is distributing their precious intelectual property of BLIDs).
- HTTP 1.1 keep-alive and pipelining on non-blocking TCP sockets.

Other notes:
- There is NO support for superlots or tier-based prices.
- There is NO support for multiple lots of identical items (same ID, same color, same condition). Please don't do that. I'm not sure what features will break exactly, but bad stuff will happen.

Downloads:
Linux AMD64/x86-64 64 bits: http://www.rayforce.net/bricksync-linux64-003.tar.gz
Windows XP and up 64 bits: http://www.rayforce.net/bricksync-win64-003.zip
I'll produce OSX binaries as soon as I can get access to such a machine to compile the software (installing a cross-compilation GCC is rather painful). Also, let me know if anyone still uses a 32 bits operating system.

Note that the source code isn't provided with this first release. I'm keeping all options open as I haven't yet given much thought how to proceed... If anyone is curious, it will remain free and donation supported for all small and medium sellers, but I'm thinking of perhaps requiring registration for the big commercial sellers? And if the total of donations & registration reaches some amount (a fraction of what 200 hours would have paid at the day job), it would then be released as open-source for anyone to use and improve. Meh, I'll figure all that later, but it's worth mentioning that this version will refuse to synchronize inventories above 250000 parts.

I'll be waiting bug reports. :)

Stragus
«134567

Comments

  • 325 Comments sorted by Votes Date Added
  • OMG You are awesome!!! I will keep you updated.
  • edited September 2014 Vote Up0Vote Down
    Oops... I wish I could edit the post above!

    The statement "Also useful to "return to inventory" a cancelled order." should appear next to the add command, and clearly not next to the sub command. Note that you shouldn't return items to inventory from within BO or BL for cancelled orders, or BrickSync will just take them out as soon as it sees the inventory mismatch.

    BrickLink's multiple order batches and item removal requests *should* be handled correctly, but that part could really use some more testing.
  • Hello,

    this looks very useful, good job.

    Assuming I part out a set on BL will BS update my BO inventory or are all non matching pieces deleted?

    Christian
  • Sounds great, but unfortunately this one is a deal killer for me...

    - There is NO support for multiple lots of identical items (same ID, same color, same condition). Please don't do that. I'm not sure what features will break exactly, but bad stuff will happen.

    I do have duplicates for various reasons. The only reason out of my control is that the new "MOC Shop" program creates duplicate lots that are "reserved" for the inventories of MOCs for sale. These are denoted ONLY by something in the remarks field on BL. You may need to warn users that they can't be MOC shop participants. Would it be possible to extend the unique key to include the lot ID#? Or maybe the multi-key be expanded to part/color/condition/comment/remark.

    Additional questions assuming the above could be resolved:
    1. Is there an ability to "filter" a partial sync? For example, I have decided not to sell any used parts on BO. So only condition=new is currently synced.
    2. What is the mechanism to detect new orders on either side? is it checking on a timer, and if so does the timer count as one of the limited API calls?
    3. I have +/- 14,000 lots in my inventory. Would an initial sync or a deep sync be possible with the 5000 limit?

    -Jason
  • 4. It's possible on either side to change a part #, a color, and a condition of an existing lot. The only true record key is the lot ID. Part numbers change due to catalog admin updates (there is a hidden UID behind a part number - its a BOID on BO, not sure what it is on BL but its there.). Is there a risk in having a user or catmin update break a lot sync?
  • Anyone who has downloaded the software before this post, please download again to fix an important bug.
  • Anyone who has downloaded the software before this post, please download again to fix an important bug.
    copy that
  • Assuming I part out a set on BL will BS update my BO inventory or are all non matching pieces deleted?
    Non-matching parts will be deleted once detected.

    Here's a recommended sequence to part out sets:
    1) Make a list of new parts in BrickStock, by parting out set(s) within the software if desired.
    2) Save that BrickStock inventory, let's assume the file is named "MyNewStuff.bsx" and located in the same directory as bricksync.exe
    3) In BrickSync, type "sort MyNewStuff.bsx". Your .bsx file is now filled with comments/remarks that match the tracked inventory, so you know where parts ought to be physically stored (assuming that information is found in the comments/remarks fields).
    4) Open the updated .bsx file in BrickStock again. From the comments/remarks, combine all parts with your existing physical inventory. You will also want to specific prices and comments/remarks for new lots (that would be the ones with blank fields).
    5) Save that inventory in BrickStock, let's say the file is named "MySortedStuff.bsx" this time.
    6) In BrickSync, type "add MySortedStuff.bsx.

  • edited September 2014 Vote Up0Vote Down
    I do have duplicates for various reasons. The only reason out of my control is that the new "MOC Shop" program creates duplicate lots that are "reserved" for the inventories of MOCs for sale. These are denoted ONLY by something in the remarks field on BL. You may need to warn users that they can't be MOC shop participants. Would it be possible to extend the unique key to include the lot ID#? Or maybe the multi-key be expanded to part/color/condition/comment/remark.
    Right, there's another feature I forgot to mention: all lots where the character '~' (without the apostrophes) is found in the remarks field will be ignored by BrickSync (example: remarks of "B314" can become "B314~"). BrickSync won't see and won't touch these lots.

    The BlLotID is indeed preferred for matching items, but the code falls back on BLID/BlColorID/Condition if it isn't available (I need to double-check if there are situations where this can occur...). Apart from that, when adding new inventory, it will also combine with the first lot matching BLID/BlColorID/Condition, which may not be what is desired if there are multiple identical lots. If needed, options for the add/sub/load* commands could specify to include comments/remarks into the lot matching requirements.
    1. Is there an ability to "filter" a partial sync? For example, I have decided not to sell any used parts on BO. So only condition=new is currently synced.
    The '~' flag above should do the trick, though I see being able to add filtering conditions could be useful. Filtered items should be managed on BL or BO the traditional way.
    2. What is the mechanism to detect new orders on either side? is it checking on a timer, and if so does the timer count as one of the limited API calls?
    Yes, it's timer-based polling and it counts as one API call. Though I realized something... I know you catch incoming order emails; I'll add something to trigger a "check" on demand from external software, such as sending a SIGUSR1 signal to the process, killall -s SIGUSR1 bricksync (and the equivalent on Windows, whatever that would be).
    3. I have +/- 14,000 lots in my inventory. Would an initial sync or a deep sync be possible with the 5000 limit?
    The "sync" operation computes the difference and only apply that difference. If your two inventories already match, there will be no operation performed, for a total of 3 API calls. You'll need an alternative to the loadprices command though, some manual upload of a bunch of XML files.
    4. It's possible on either side to change a part #, a color, and a condition of an existing lot. The only true record key is the lot ID. Part numbers change due to catalog admin updates (there is a hidden UID behind a part number - its a BOID on BO, not sure what it is on BL but its there.). Is there a risk in having a user or catmin update break a lot sync?
    Good point. That is probably not an issue, as matching items by BLID/BlColorID/Condition really is the fallback method, but I need to double-check this.
  • When reading all this stuff about 'synchronising' I think I'll pass.
    1. Bricklink has updated partnumbers of several 'decorated' parts, adding an 0 for example so pb001 in stead of pb01, don't see how a sync software can keep up with that, unless someone changes the BO coding, and Admin has made clear he won't spend time on it.
    2. New items with particular numbering on BL have not been matched for the past year, and the number of items added (particulary stickered and decorated parts and minifigs) between BO and BL is not the same either. BL also does not follow LEGOitem numbers, while BO does. As time moves on (month after month, year after year), the number of errors during the 'sync' will most likely increase...
    3. Value: Over time partsvalue (average) will start to derive between BO and BL, no sync software will be able to match 'average' from each side, so one will end up with overrated parts on one site and undervalued items on the other site. For example I've checked many items that I think that are important for my store, compared availability and increased (but also decreased in some cases) the value here on BO, result: higher benefit because less competition on certain items. By using a sync (same price both sites I assume) I would simply loose money.

    While I haven't concentrated much on listing the past 6 months (either site), I still had quite good returns, and running on both sites certainly made a difference, but not that much actually (after all, what you sell on 1 site, can't be sold on the other). If I count januari 1st till now, counting both together gives me about 87% of my BL sales in 2013, but still have 3 months to go (on BO, as my BL shop has been closed for over a month now), so most likely I will reach the same figure as 2013. But during a period of 8 months, I spended hours for manual sync, I decided a month ago it's not worthwhile to keep synchronising, and by spending the amount of time I used to spend on the sync, on sorting and listing and adapting prices to BO averages in stead (or higher if there is lesser competition), it might simply be more worthwhile to put all the energy in BO only. Not saying I won't be selling on BL anymore, but if I do it will be with a very limited amount of items and controlable quantity (stickers and minifigs for example as I never list full quantity).

    I'm sure a sync soft would save a lot of time, but one should take in consideration that the number of errors during a sync will slowly increase over time, then what? And as indicated certain options (superlots / tiered pricing) are not supported. Not to mention the fact you might be off track when it comes down to both sites value averages...
    Seriously, I think if people would concentrate a little more on adjusting prices by checking availability and BO averages, one might simply be better off running just on BO, it seems a lot less hassle (not to mention possible issues with stockrooms and such).
    I wish good luck to all who try to keep the sync active, but I doubt it will be satisfactory in the long run...

    Eric

  • And to support the 'average' value issue in the long run: Check the average value of sold STIG keychains between BL and BO, 6 months average difference is about 10 Euro, so a seller who 'syncs' with a software will either (potentially) loose money on 1 site and be too expensive on the other site... I prefer ending up with the best benefit ;-)
  • edited September 2014 Vote Up0Vote Down
    1. Bricklink has updated partnumbers of several 'decorated' parts, adding an 0 for example so pb001 in stead of pb01, don't see how a sync software can keep up with that, unless someone changes the BO coding, and Admin has made clear he won't spend time on it.
    2. New items with particular numbering on BL have not been matched for the past year, and the number of items added (particulary stickered and decorated parts and minifigs) between BO and BL is not the same either. BL also does not follow LEGOitem numbers, while BO does. As time moves on (month after month, year after year), the number of errors during the 'sync' will most likely increase...
    I think we should be able to keep BrickOwl up-to-date on the matter... at least for the parts we actually have for sale. Note that BrickSync uses BrickOwl for the translation of BLIDs into BOIDs, results are cached but they can be flushed if necessary (to flush the translation database, delete the file data/bricksync.trs).
    3. Value: Over time partsvalue (average) will start to derive between BO and BL, no sync software will be able to match 'average' from each side, so one will end up with overrated parts on one site and undervalued items on the other site. For example I've checked many items that I think that are important for my store, compared availability and increased (but also decreased in some cases) the value here on BO, result: higher benefit because less competition on certain items. By using a sync (same price both sites I assume) I would simply loose money.
    You assume correctly, though it would be relatively easy to implement different prices for BrickLink and BrickOwl. I'm not sure I want to add that feature though, as I don't think BrickOwl would benefit...
    Not to mention the fact you might be off track when it comes down to both sites value averages... Seriously, I think if people would concentrate a little more on adjusting prices by checking availability and BO averages, one might simply be better off running just on BO, it seems a lot less hassle
    Agreed, but I must say this software wasn't exactly designed for existing BrickOwl sellers like you :p. Rather, this is meant for BrickLink sellers who are reluctant to move to BrickOwl due to the perceived risks (real or imaginary), but surely wouldn't mind selling on both marketplaces simultaneously. That being said, I think we should wait until we have had some more in-house testing before posting about the software elsewhere.

  • You assume correctly, though it would be relatively easy to implement different prices for BrickLink and BrickOwl. I'm not sure I want to add that feature though, as I don't think BrickOwl would benefit...
    Brickowl would not benefit (allthough it would slightly because of fees), any seller would.
    another example: Part out value of set 10232 (new): BL 242.14 (100% of items) BO 273.62 (maximum 89% of items), so return on BO is simply better. By doing a sync based on BL averages, a seller parting out that set would be undercutting any other seller, as a matter of fact, the seller would be undercutting him/herself, as by simply doing a 'BO' partout he/she would be better off.
    Try this with 10 different sets and the result will most likely always be in favor of BO.
    Dunno wether people noticed, but items seem to sell for higher prices on BO, so why waist time (even if it's automated) to sync or why have less benefit :? I've always gone by the fact that when I list something it will sell sooner or later, wether it's on 1 site or on 2 (and you can't sell it twice), the result is the same in the long run.
    Agreed, but I must say this software wasn't exactly designed for existing BrickOwl sellers like you :p.
    Rather, this is meant for BrickLink sellers who are reluctant to move to BrickOwl due to the perceived risks (real or imaginary), but surely wouldn't mind selling on both marketplaces simultaneously. That being said, I think we should wait until we have had some more in-house testing before posting about the software elsewhere.
    Oh, thanks, now I feel excluded from this wonderfull sync tool LOL
    No, seriously, this site is growing rapidly with buyers who are (clearly) ready to spend decent amounts of money, spending energy to have this site grow more and faster seems far more valuable then to keep running on both (particulary as the competition on BL is quite high in 'number' of sellers). Any seller who takes a peek at prices should simply be smart enough to deduct it's more worthwhile to sell on BO. The only issue BO has is that's it's database is far from complete, if people would stick to BO and even by selling a little less at first, but spending some free time to add data (pictures for example), it will be to the advantage of all sellers here. Right now people are selling on both, they don't have the time to work on the database (imagine how much you could have done during the time you spend on this sync stuff) and they feel the need to sell on both sites to maintain their 'flow'. Taking away the need (= increasing the flow on BO) by selling a little less and spending some time on BO data seems a much better choice in the long run, but it seems most sellers here fail to understand that...
  • edited September 2014 Vote Up0Vote Down
    Right now people are selling on both, they don't have the time to work on the database (imagine how much you could have done during the time you spend on this sync stuff) and they feel the need to sell on both sites to maintain their 'flow'. Taking away the need (= increasing the flow on BO) by selling a little less and spending some time on BO data seems a much better choice in the long run, but it seems most sellers here fail to understand that...
    I'm all for improving BrickOwl's data, though my personal opinion is that writing software for BrickOwl is also time very well spent. The synchronization software is the first step, the next step would be BrickStore-style software, OpenGL-based to run on any device (desktop, tablet, phone, etc.). Among other things, it should be able to connect to BrickOwl or to BrickSync, possibly running on a different computer, in order to manipulate the real inventory directly.

    Besides, you are already doing a fantastic job maintaining the catalog. I don't get any point for writing software. :p
  • I sure hope for your sake no-one will consider your sync software a method to download displayed data (including ones inventory) for which a breach of a certain ToS might be considered and lead to full deletion of data or worse... and anyone using such soft would be most likely in the same position.
    Maybe something to take in consideration...

    I'm not 'maintaining' the catalog, Admin Lawrence is, I'm just adding data, it generates points yes, but those have no effect on my store. I would rather see 50 other people doing the same thing so that in the long run (but preferably soon as possible) it has a real effect on my store (and the store of any other seller), by selling more items (and not just the standard stuff), like I said it seems most sellers fail to understand that it is exacly the 'missing data' that is preventing to generate more and nicer orders.
    And to be all honest, I'm slowly getting to a point to decide to become really selfish and only update data that is usefull for my own store (and therefor generate an immediate effect on my sellings) in stead randomly providing data for the benefit of all, as right now the vast majority of sellers are not contributing whatsoever. What do they expect, that all will be done by Admin and handfull contributors? At this pace it's gonna take years to fully update the catalog and on the other hand they will be saying: wow, I need to keep both my stores to maintain the same flow because BO isn't generating 'enough', while a bit of effort from all would have an immediate effect on the overal growth (i can tell from certain items I provide pictures for and then sell at good prices), most likely leading to a point where a 'sync' becomes totally useless because one could simply stick to selling on BO only and still have loads of orders.

  • I sure hope for your sake no-one will consider your sync software a method to download displayed data (including ones inventory) for which a breach of a certain ToS
    As far as I know, BrickLink's Terms Of Service allows accessing the API for one's personal inventory management needs. What is forbidden is having a third-party manage the data, but there is no third party involved since the seller himself manage his own inventory on his own computer. This is very similar to the private software other sellers have written for their own needs.
    And to be all honest, I'm slowly getting to a point to decide to become really selfish and only update data that is usefull for my own store (and therefor generate an immediate effect on my sellings) in stead randomly providing data for the benefit of all, as right now the vast majority of sellers are not contributing whatsoever. What do they expect, that all will be done by Admin and handfull contributors?
    While I agree that we need more catalog contributors, I don't think it's appropriate to blame me for the time spent on BrickSync. Come on, writing this software is also time well spent, it will be beneficial to BrickOwl and its users.

    But let's try to keep this thread on the topic of BrickSync, catalog contributions are a complex topic... I really do appreciate the work you are doing though (in fact, I think there should be a way to donate to the catalog contributors, with the total amount donated to each one being displayed so that we can be fair to all of them).

  • As far as I know, BrickLink's Terms Of Service allows accessing the API for one's personal inventory management needs. What is forbidden is having a third-party manage the data, but there is no third party involved since the seller himself manage his own inventory on his own computer. This is very similar to the private software other sellers have written for their own needs.
    Well, I sure hope you're right but the terms are formulated in a different way: it is 'upload, or transfer to any third party' read the TO...
    BO is a third party and if you use software to upload TO a third party, then BO can't be blamed, but any 'user' doing such might be blamed. I'm not a lawyer, but I think the phrasing they did was to prevent 'any form' of upload. Before actively spreading around such a software, you better inform yourself at a lawyer.

    While I agree that we need more catalog contributors, I don't think it's appropriate to blame me for the time spent on BrickSync. Come on, writing this software is also time well spent, it will be beneficial to BrickOwl and its users.
    Oh dear, I'm not blaming you whatsoever, if that was your impression, then pardon me, not intended. One is free to do in their own free time whatever they like and whatever they find usefull, I was just speaking in general about the vast majority of people not 'contributing' . When it comes down to sync soft, like I stated, it might simply not be of high need if the catalog was 100% OK, because then people could stick to selling on just one site (like hundreds of sellers stick to BL or Ebay or Amazon only and manage just as well). I just see a lot of people eager to sync, in stead of concentrating helping to build the data here (but who am I kidding, the same thing always happens on BL just as well). Besides that I think it's nobel you try to do something that might help people (or yourself) to have an extra option, but from my point of view such would be useless if people would help build the catalog (and so all would be helping all)
    But let's try to keep this thread on the topic of BrickSync, catalog contributions are a complex topic... I really do appreciate the work you are doing though (in fact, I think there should be a way to donate to the catalog contributors, with the total amount donated to each one being displayed so that we can be fair to all of them).
    If people are too shelfish to not contribute, I doubt they will think otherwise about donations :D
    Besides, technicly 'points' lead to prizes, wether those are/will be enough to compensate decently the effort of the contributors is another question (and up to Lawrence), but not really part of the topic indeed. Maybe fees should be raised and then part could be spreaded among the contributors :P

  • I would rather fix these known bugs than posting about them, but I'll be away from the computer for a few hours.

    Known bugs for 0.04:
    - Uploads to BrickOwl presently fails for "Packaging"/"Box" item types.
    - Some bug related to having "Unsorted lots" in your BrickLink inventory.

    Fixed bug in 0.04:
    - BrickLink PURGED orders are handled properly.

    Thanks PSBricks!
  • Not sure if this is a bug or what but....

    I have 5 items on BL, 3 colors of 3001, the tumbler and the mini cooper. I used the sync to get a inventory .bsx. I open the file in brickstock only to have the 2 sets error out and delete. Not pulling the number just the description.
  • Is your BrickStock database up-to-date? If it asks to delete the items, it's because BrickStock isn't aware the items exist, and a database update should fix that.

    BrickStock should really just leave unknown items alone, but eh, I didn't write that software.
  • edited September 2014 Vote Up0Vote Down
    Version 0.05 uploaded to the links above (even though the links and file names still say "003").
    - Fixed some bug related to BrickOwl uploads of Packaging/Boxes. It turns out BrickOwl doesn't agree packaging can be "new" (fine, that makes sense).
    - Fixed some bug related to having Unsorted Lots in the BrickLink inventory.
    - Added handling of SIGUSR1 signals (Unix only) to externally trigger a "check" on both services. I'm not finding anything similar for Windows, I guess we could use something crude like creating a file "bricksync.check.now" in BrickSync's current working directory...
  • Version 0.06 uploaded, fixing a bug related to zero or negative quantities in certain circumstances. The automatic "sync" that follows the warnings/errors would correct any issue, but I still recommend anyone running BrickSync to upgrade.

    Thanks for the bug reports, keep them coming. :D
  • Version 0.07 uploaded, fixing an issue related to multiple BrickLink batches for a same order.

    I'm finally beginning to think the software should have a way to check for new versions on its own! :)
  • edited September 2014 Vote Up0Vote Down
    Version 0.08 uploaded: fixed stuff related to items for which the BOID lookup fails, and using valid "condition" codes when creating new lots of sets on BrickOwl.

    I'll improve the BrickOwl item conditions later, to properly handle set completeness or sealed status, and guess the grade of "used" for parts by looking for keywords in the comments field.
  • Anyway to get this on github for better version control and group edits?
  • @EricHampy I'm doing version control locally, I'll think about the way forward after moving out of this beta testing and rapid bug fixing stage. All the various issues currently popping up take edits of 2-3 lines to fix, like this one:

    Version 0.09 uploaded: The code was trying to specify a color zero (Not Applicable) when creating new lots of stickers during the "init" stage, and BrickOwl didn't like that. Fixed.

  • First step: Unpack the package somewhere accessible, there's no installer.

    Second step: Modify the file data/bricksync.conf.txt and enter API keys for both BrickOwl and BrickLink.

    Third step: Run bricksync, it will read and compare your BrickLink and BrickOwl inventory. It will ask your permission to apply all changes required to make the BO inventory match the BL one.
    Alexis, I have a question about the third step.
    Let's be clear, I'm not gonna use the program to 'sync', but I can see a certain value to step 3 for myself (and current inventory status of both my shops). Does bricksync, after comparing data between BO and BL, tells the user exactly 'which' differences? I mean does it generate a list saying something like: mismatch: part xxxx: BO quantity=5, BL quantity=17 and vice versa? I would be interested in knowing that, because at a certain point my inventories diverted as I parted out different sets but each to a specific site. What I want to know is what I 'might' have on BL that is not listed on BO so I can add those on BO or update quantity if it turns out the quanity differs (manual change, as I would need to re-count to make sure). At least a list would be quite handy to narrow down: I have 9300 lot's in my BL store, but about 1300 are in stockroom and I cannot risk moving those, as those lot's are my personal collection, so if I can weed out those and weed out identical lot's with identical quantity, it would be a massive help in stead of comparing 9300 lot's with 7900 BO lot's.

  • Does bricksync, after comparing data between BO and BL, tells the user exactly 'which' differences? I mean does it generate a list saying something like: mismatch: part xxxx: BO quantity=5, BL quantity=17 and vice versa?
    Yup! It will only print a summary before asking for your permission to proceed, but if you inspect the log file in data/logs/, it will list exactly every single change (even if it's just a change of item comments). Note that your BrickLink store has to be open for this to work; a closed BrickLink store returns an empty inventory.

    Also, version 0.10 uploaded:
    - New command "owlresolve" to perform BLID->BOID lookups for all items from the local inventory which are still lacking BOIDs. It will list all items for which the BOID lookup fails (now go edit the BrickOwl database and fill them in :p). If new BOIDs were resolved, typing "sync brickowl" will upload these missing items to BrickOwl.


  • Let's be clear, I'm not gonna use the program to 'sync'

    Eric, just a hint: you might want to reconsider that, and look into it. I am using (not the program from Alexis, but I will look into that) a sync program as well, and I am really, really glad I made that choice.

    Of course it is your decision, just wanted to let you know that it actually works, and can save a lot of time.

    So Alexis: Thanks for making this public, I will look into it this weekend.
  • @EricHampy I'm doing version control locally, I'll think about the way forward after moving out of this beta testing and rapid bug fixing stage. All the various issues currently popping up take edits of 2-3 lines to fix, like this one:

    Version 0.09 uploaded: The code was trying to specify a color zero (Not Applicable) when creating new lots of stickers during the "init" stage, and BrickOwl didn't like that. Fixed.
    Just getting confusing you say 0.09 is download but the download still says like 0.03.
  • edited September 2014 Vote Up0Vote Down
    Just getting confusing you say 0.09 is download but the download still says like 0.03.
    I know! I should have foreseen that I wouldn't be able to edit the first post ever again... (Admin, any hope of being able to edit that post?)

    Version 0.11 uploaded:
    - When uploading sets, the code will now make use of the "completeness" value if known (otherwise, it assumes new sets are sealed and used sets are complete).
    - When uploading parts in an used condition, BrickSync will now apply some heuristics to guess if the BrickOwl grade should be "like new", "good" or "acceptable".

    The heuristics evaluate what's in the item's comments field to make their guess. It recognizes some basic vocabulary (scratches, new, excellent, good, discolored, faded, broken, dented, etc.), along with modifier keywords amplifying/diminishing/inverting the expression depending on context (very, small, deep, few, no, not, etc.). Simple compound statements such as "no deep scratches" or "not broken" should be properly evaluated, or perhaps even "not very few not deep large scratches". :D

    "There is no 'overkill.' There is only 'open fire' and 'I need to reload.'" - Maxim #37
  • @Stragus - really appreciate you doing this - that is one heck of an undertaking and for $0.00 ?

    Makes thee a star!

    Being a Apple user I guess it is to be limited to Windoze/Linux?

    If that is so - I may boot my Mac in Windoze mode - this is not the emulator software - it literally boots in Windows - will the software function correctly?

    I no longer sell parts on BL, however I would really benefit from full integration between BrickOwl and BrickStock - can I use it for that purpose? ie export BrickOwl inventory to Brickstock, edit as required and load back to BrickOwl with changes?

    Thanks! Graham
  • @Stragus - really appreciate you doing this - that is one heck of an undertaking and for $0.00 ?
    I won't refuse donations after it moves out of beta testing :). And it's currently limited to 250000 parts to keep possibilities open regarding some kind of registration for big commercial sellers, I'll figure that out later.
    Being a Apple user I guess it is to be limited to Windoze/Linux?
    No no, it would run fine on OSX. I need to ask around to find a friend running Apple on a static IP, to connect remotely and compile the software for OSX whenever needed. Installing a cross-compilation environment is tiresome, that would be the fallback solution...
    I no longer sell parts on BL, however I would really benefit from full integration between BrickOwl and BrickStock - can I use it for that purpose? ie export BrickOwl inventory to Brickstock, edit as required and load back to BrickOwl with changes?
    Very good point. The current software wouldn't work for that purpose. Let me figure out what would be required for a BrickOwl-only mode of operation.
  • Thanks Stragus (*) (*) (*) (*) (*)

    and I will be more than happy to donate for this service :)

    Graham
  • Anyway to get a little like step by step on adding or adjusting items after we start up the service.
  • Version 0.12 uploaded:
    - IMPORTANT BUG FIX Fixing an important bug regarding BrickLink order handling, see below.
    - Added code to properly handle BrickLink's inventory completeness of "X" for sets, totally undocumented but it comes up.

    Due to (misguided?) concerns for the limit of 5000 API calls/day, I specified that very old BrickLink orders shouldn't be downloaded during an init from scratch. If these old orders are later changed (like a buyer marking it as "complete"), it computes the "difference" of the order's current inventory from the previous one (which we don't have, therefore an empty inventory). That results in a bunch of incorrect inventory changes. The damage can be reversed from the backups and commands, but it's still a mess.

    To anyone updating the software: only replace the executable file, always keep everything else in data/ (state, inventory, etc.). BrickSync will always be able to read the files generated by previous versions.
  • edited September 2014 Vote Up0Vote Down
    Anyway to get a little like step by step on adding or adjusting items after we start up the service.
    Sure! I'm sorry I was more focused on finding bugs than making a friendly guide so far, but we can do both. (The last bug was a shortsighted one, darn it; totally the wrong solution after someone expressed concerns over the count of API calls for fetching all BrickLink non-purged orders during Init.)

    See this image for a graphical example of how to add inventory as a BrickStock .bsx file. I then remove the lot with the setquantity command:
    http://www.rayforce.net/addinv2.png

    Or see this example of editing an existing item's remarks:
    http://www.rayforce.net/editremarks3.png
  • edited September 2014 Vote Up0Vote Down
    Let's also try a verbose guide how to part out a set with BrickStock and BrickSync.

    Step 1: I create a list of parts in BrickStock. Let's assume I'm parting out a 70709 Galactic Titan, so I save the inventory as "70709.bsx" in BrickSync's directory. I now close that file in BrickStock.

    Step 2: To know where to put the parts in the physical inventory, I need the comments and remarks fields to match what's in the store's inventory, so I use the sort command:
    sort "70709.bsx"
    http://www.rayforce.net/partoutset.png

    Step 3: I open the file "70709.bsx" back in BrickStock. For every lot that existed in the store's inventory, all comments, remarks and prices have now been updated. Parts which don't exist in the store's inventory are not modified. On top of that, it's sorted by Color and then by Name to quickly locate parts:
    http://www.rayforce.net/brickstoresort.png

    Step 4: I combine the parts where the remarks field tells me that I have an existing such lot. When new lots are encountered, I'll want to enter new prices, remarks and prices. I save the new file as "70709sorted.bsx".

    Step 5: My file "70709sorted.bsx" is ready to go. If I type add 70709sorted.bsx, all quantities for existing lots will be merged. New lots, for non-existing items, will be created with all fields defined.

    Step 6, optional: The previous step, using the add command, would not change comments, remarks or prices for items that already exist in the inventory. If I intend to update them with new values found in the file 70709sorted.bsx, I should type:
    loadprices 70709sorted.bsx ... to update prices for all lots from the file 70709sorted.bsx
    loadnotes 70709sorted.bsx ... to update comments/remarks for all lots from the file 70709sorted.bsx
  • Version 0.13 uploaded:
    - Fixed Windows-only issues related to some standard functions (snprintf(), vsnprintf(), etc.) not actually being standard-compliant on Windows (what else is new!). A symptom was skipping updates for BrickOwl comments/remarks in some circumstances (to fix everything, update executable and type "sync brickowl").
  • Anyway to get a little like step by step on adding or adjusting items after we start up the service.
    Sure! I'm sorry I was more focused on finding bugs than making a friendly guide so far, but we can do both.
    Well I meant whenever you get the time. Didnt mean like you must stop everything and do it right now. But ok. If more people can properly use the script then the more bugs will be found. Developer 101.
  • I have finally obtained remote access to some mac-mini to compile OSX executables.

    Hence, this shall be known as the link to download up-to-date OSX binaries, obligatorily pursuing the absurd convention of naming the package "003" due to my inability to edit the thread's first post ever again. :)
    http://www.rayforce.net/bricksync-osx64-003.tar.gz

    Also, version 0.16 uploaded:
    - Fixed some Unicode/UTF8 conversion during JSON encoding and decoding. If you had Unicode characters in your comments/remarks and noticed unnecessary BrickOwl updates during "sync" when the comments/remarks already matched, that was the issue.
    - Fixed an old issue where a large queue of BrickOwl updates could pause after some read/write errors, to resume 30 seconds later. The actual issue was ignoring the server's HTTP Keep-Alive "max" parameter specifying a maximum count of pipelined HTTP queries on a same socket.
    - Inventory backups are now also stored before every add/sub/loadprices/loadnotes command.
  • I am sure I am doing something wrong but...

    I place my BL/BO keys in between the "" marks correct? Then open brick sync.

    Take a look at the attached image for what I am getting. (I blocked out my computer name in the pic for my own security reasons)

    Yes I put them in the data folder in the bricksync.conf file and saved it.


    I am by no means a programer at any level, so I am positive it is my doing here. But any help?
  • edited September 2014 Vote Up0Vote Down
    @budgetkids The first step would be to open one of the 3 saved errors, data\errors-2014-09-29\00000.txt and see what the BL server is saying. There's probably something wrong with the credentials... I suggest creating the token for IP and IP mask of both 0.0.0.0

    And the software could print the error message besides saving it, I'll agree on that!
  • edited September 2014 Vote Up0Vote Down
    Perfect. Thank you for the help. Yea, the ip was all "0" but I left the IP mast where it was at, which was 255's.
    Issue resolved. now cross my fingers and hope I don't mess my inventory up.
  • Ok. I think everything worked ok. There are errors but yet it says complete? I attached a pic with an arrow.

    Also, I want to make sure I understand the program before I really get into it and mess things up.

    1. If I add inventory to BL the program will sync the new lots/updated quantities to BO correct?

    2. If I get a BO order, will it deduct the items from my BL inventory or is this something I still need to do?

    3. It looks like some BOIDS were resolved, but some were not. Is the "BOID Resolution" system you have created a work in progress or are some BOIDS never going to be resolved with BL ID's?]

    - There is NO support for multiple lots of identical items (same ID, same color, same condition). Please don't do that. I'm not sure what features will break exactly, but bad stuff will happen.
    4. I have duplicate lots in my inventory with the same price. When I part out a set. then part out another set at a different time, they may have the same items in it, which would make duplicate lots on BL. Are you saying not to use your program until I get these lots merged? (I just saw this...after I used the program... Glad I made a back-up before I used your program in case something bad does happen)

    Ok, I think thats about it. Oh, maybe have a feature to first merge exact duplicate lots if we want? Same condition, price, item number, color then merge those as a commend maybe?


    Chris
  • SO sorry for the multiple posts.

    I just got BO order. i see program did update BL successfully.
    HOWEVER I do see the the lots do not get retained, BrickSync completly deletes the lot instead of editing the quantity to zero and stockroom it.
  • Ok. I think everything worked ok. There are errors but yet it says complete? I attached a pic with an arrow.
    Right, that's just the underlying TCP SSL code complaining about BrickOwl unexpectedly closing a socket on us :), but it reconnected and completed the job. I should probably hide these messages... When in doubt, type "sync" and it will verify both BL and BO for any difference with the tracked inventory. The slightest difference will be corrected.
    1. If I add inventory to BL the program will sync the new lots/updated quantities to BO correct?
    Negative on that, you should stop editing inventory through BL or BO (unless items are flagged by '~' in remarks not to be synchronized). If you want to add inventory, use the command add MyBrickStockFile.bsx. Also, see the post above about a "verbose guide how to part out a set with BrickStock and BrickSync".
    2. If I get a BO order, will it deduct the items from my BL inventory or is this something I still need to do?
    Nothing to be done, order inventories are managed both ways.
    3. It looks like some BOIDS were resolved, but some were not. Is the "BOID Resolution" system you have created a work in progress or are some BOIDS never going to be resolved with BL ID's?]
    It uses BrickOwl's database for BLID->BOID translation, these BLIDs are just unknown to BrickOwl. You can type owlresolve to print a list of all missing BOIDs... then go fill in the blanks in BrickOwl's database :).
    4. I have duplicate lots in my inventory with the same price. When I part out a set. then part out another set at a different time, they may have the same items in it, which would make duplicate lots on BL. Are you saying not to use your program until I get these lots merged? (I just saw this...after I used the program... Glad I made a back-up before I used your program in case something bad does happen)
    I think everything will be fine, as the software primarily relies on LotID for identification and BLID:Color:Condition is the fallback path. But I need to double-check all of this. And the "add" command will merge stuff with the first lot matching BLID:Color:Condition it finds.
    Ok, I think thats about it. Oh, maybe have a feature to first merge exact duplicate lots if we want? Same condition, price, item number, color then merge those as a commend maybe?
    Hum yes, agreed on that. I'll add a "consolidate" command later today.
    I just got BO order. i see program did update BL successfully.
    HOWEVER I do see the the lots do not get retained, BrickSync completly deletes the lot instead of editing the quantity to zero and stockroom it.
    Right. I didn't think someone could want to keep lots in stockroom (never did that myself), but it's worth adding an option for it. Thanks.
  • Thank you so much for such a quick reply.

    Normally I wouldn't retain the lots over there, but sometimes pictures can be hard to see here so I use the lot ID to make sure I grab the correct item on BL to see their picture. I mainly do this for torsos and some heads/helmets. Getting a different type of picture helps re-assure my wife and I we are packing the correct item.

    Again, THANK YOU SO MUCH FOR THIS!
    Chris
Sign In or Register to comment.