Personal part repository - part development process

Hi,
I’m doing several kind of parts, some are really personal other could be shared but not finished.

I code parts as XML/SVG for many (maybe stupid) reasons. Once a part file is modified (fzp or svg), to test it, I would like to publish it to Fritizing.
I try to modify the files in the fritzing/part/user folder but it doesn’t seems to be the right approach.

I wondering if it can be possible to push my files in a repository (like github, git, svn or a folder or …) and to use the automatic part update of fritzing ?

Maybe you also have additional advice about the development process…

I guess by publishing to Fritzing you mean using the part in the Fritzing program?
I have no advice, only a little experience to offer. Probably the pro’s of the forum have some real advice.

The last part I did, I did in an XML editor, directly editing the FZP and SVG files. I did edit the files in the parts/user folder where the parts from the “Mine” bin are stored.

I’m not sure what problems you encountered, that make you say it’s not the right approach. In my experience, Fritzing will pick up changes to a part that you make in the parts/user folder. But if you have problems with the part refreshing, you need to drag a new instance of the part from the bin to the sketch. I’m not sure if the part already in the sketch did update.

If by ‘publish’, you mean share with the community, you can post the .fzpz files here. There is a “Parts Submit” category. You can also put the parts (and/or) the fzp and svg files into a github (or other) repository, and post a link to that here.

If you are editing the fzp and svg files directly in the user folder, you can (re)create the fzpz file by doing an export of the part from the “Mine” bin. My process is to create (and work on) part files outside of the Fritzing folders, create the fzpz from that, then open the fzpz from Fritzing, which will import it into the mine bin and user folder. With limitations based on the current and existing part module id, and files. You may need to delete any existing (matching) part from the mine bin before Fritzing will load the new one. That does NOT affect the part in an existing sketch. It will still have a copy of the older part, until it is explicitly updated/replaced.

None of that will get access to the Fritzing automatic parts library update. That only applies to parts in the Fritzing part library repository (the “core” parts). To get new parts there needs a standard github pull request which, once approved, will add the part(s) to the repo, and the next update will see them.

If you choose not to use a github repository for the parts you can create an fzpz file by the following (starting from a part in repository format):

rename the fzp file to part.name.fzp

test.fzp -> part.test.fzp

rename each of the 4 svg files to svg.layername.part.svg

test_1_breadboard.svg -> svg.breadboard.test_1_breadboard.svg

same for icon schematic and pcb svgs. Move them all to the same directory and zip (I use 7zip) the 5 files in to an fzp file:

part.test.fzp
svg.breadboard.part.svg
svg.icon.part.svg
svg.schematic.svg
svg.pcb.svg

as

part.fzpz

Now you can import (via open->part.fzpz in Fritzing) or post here in the forum. The Fritzing loader program converts the part. etc files in to repository format in an internal directory (the mine parts bin.) As far as I know this is not officially documented anywhere.

Peter

Thanks @vanepp @microMerlin

What would be the best development and test process ? should I modify the files in parts and svg folder or should I always go through the part-editor ?
what regenerate database means ? does it reimport/rebuild a cache database from svg/fzp files ?
should I have a dedicated install of fritzing to test ?
How do you do ?

I don‘t think there is a best process, only a best process for you. I believe you will find that everyone here does it a bit differently. In the end you will have to settle for a process that suits you best.

But I agree, while trying to figure out what that is, it is interesting to hear how different people work, who have a lot of experience.

Where did you get the „regenerate database“ from?

Parts Editor can only ‘edit’ the fzp file content. It has no ability to make changes to any of the svg files. All it can do is import a new image that has been created or changed using some other tool.

Modifying fzp files directly in the Fritzing user folder is a bit of a ‘risk’, in that it is quite possible to break a part definition with an incautious change, that Fritzing might not notice (until much later), because it did not see it during the import process. I do not know how much actually gets looked at during the import. It might be just what the svg files are, and that they match the defined view paths. The only ‘requirement’ for the import, is to unzip the fzpz file, put the contained fzp and svg files in the correct folders, and check if the module ID is already being used by an existing part. It might do more, which would be bypassed by direct edits.

Due to that (lack of knowledge), I prefer to always do editing outside of the Frizing parts.user, parts.svg folders, and let Fritzing do the import of any changes. Though that means that the existing part has to be deleted from the bin first, to prevent conflicts with Module Id (unless that is changed as part of the edit). If you are doing the changes with Parts Editor, that does not apply. It will be doing the work directly on the files in the parts sub-folders, but it can also do whatever validation is appropriate.

I Agree that there is no single ‘best’ process. For ‘my’ process, I have a folder I use for checking (not usually changing, except as part of a test) parts. That has a notes file with information I find useful during testing. That includes content templates to use for reporting results when checking one of the parts submitted here. That folder contains 2 subfolders, one I use to run FritzingCheckPart, the other to inspect the raw fzp and svg files for a part.

For working on a part, I create a new folder. If the starting point for the new part is an existing core part, I create a copy of the part in Fritzing, export it, then unpack the fzpz into the working folder. I do not save the copy of the part in the “Mine” bin. If the base is an existing custom (user) part, I start from the fzpz for that (or recreate it as an export from the bin). Otherwise, I take sample and template files from various places, depending what the new part needs, and what I have (and can find). Initial export, and later import, is the only thing I normally use Parts Editor for.

After creating an initial version of the part files, I zip them into a fzpz, switch to the testing folder, unzip, and run FritzingCheckPart. Repeat until get a mostly clean report (and all remaining warnings explained), then import into Fritzing for visual checks and testing.

Repeat some more, Possibly first deleting a conflicting part. Usually there are no conflicts, because I do not save the ‘test’ version of parts in the Mine bin, and one of the initial changes is to set a new module id in the part. Sometimes a Fritzing (or computer) crash leaves pieces behind that need to be cleaned up manually.

With this, there should not be any need to run the “regenerate parts database” command. The import should do everything that is needed. Editing fzp or svg files directly in the parts sub folders “might” needed a regenerate. I do not know everything that it checks and builds indexes for. That command is intended, as far as I can tell, to rebuild the cached information after the core parts library has been updated. So after a the Fritzing has downloaded core part updates from the github repository.

The above is for working with “custom” parts. The process is different for making changes that are intended to be incorporated into the core parts repository. For that, I have a separate copy of the parts repository, or at least a separate branch, in a fork of the repository. Files are edited ‘in place’ in that branch, and testing is done by pointing Fritzing to that parts library. Changing to a different parts library is another reason to use the regenerate command. The other differences for changing the core parts library are mostly around the requirements of the git process of tracking changes. I still use a separate folder to run FritzingCheckPart against a copy of the new and changed files, but that is a bit more involved. It needs to duplicate the folder structure used for the core parts, but only contain the files to be checked, each in the correct folder. I normally write a small (bash) script to copy the files to where they are needed for testing. The script is created for each set of parts being worked on as a group. I had a single main folder and script when I was working on the 8 by 8 led matrix parts, though that was for 16 parts. The parts shared a lot of the same svg files in different views, in different combinations. It was really 4 different sizes, each with 2 different polarities, each with 2 possible pcb footprints.

That outlines the basic process, with minimal details about what is involved in the editing and testing.

A separate “install” of Fritzing can help in the various scenarios. But ONLY if you can manage the files that are expected (by Fritzing) to be in known locations. There is a command line flag that can be used to point at a different parts library folder, but the user configuration and user parts folders are still going to be shared. Unless that alternate install is also for a different user, or on a different computer. A different computer could also be a virtual machine. A different computer (or VM) is safest. There should be no interaction with the other install that way. I ‘accept’ the common files, an just specify a different part library when needed for basic part editing. I use a VM to be able to test using the Fritzing 0.9.5 development version.

@flaix

it’s the last submenu in Parts.

The regenerate database is only necessary after a parts update. It got mentioned lately because someone was manually updating the parts repository (because their auto update was broken) and they need to manually do that after the new repository is cloned (automatic parts update does it as part of the install.) So you can usually ignore it.

I prefer to ignore parts editor and edit the files directly. For some things (bendable leg parts for one) that is the only way to do it as parts editor doesn’t currently support it. Parts editor was not finished when development stopped so it is not complete, but many find it easiest.

Currently that isn’t easy. At present the user directory files are hard coded so all Fritzing versions will use the same files and may cause corruption. I’m poking at changing the code to allow changing the file location via a command line flag (mostly for testing new Fritzing versions) but it isn’t currently ready. I use one version of Fritzing and make sure I have backup copies of all the .fzz and .fzpz files outside of Fritzing (so I can clear the user directories if necessary.)

My typical path is described here:

Peter

@vanepp
Hi Peter,
I created some custom elements (parts, images, …). It could be interesting to specify a personal repository, github, svn, or even better an artifact repository to facilitate personal parts update from that repository as it seems we can do it for the core parts.

thanks
stephane

What do you mean by an “artifact repository”?

The update itself (of core parts) is a simple git pull from the github repo. The core parts library is a git repository. In addition though, there is code in the application that detects when parts in an opened sketch are not the latest version, and offers to update to the new version. To facilitate that, core parts (most types of changes) are not done directly to the existing part. Instead, the existing part file(s) are moved to an obsolete folder. Fritzing still finds and can use them from there. A tag is added to the obsolete part to show where to find the new version. Which is where the offer to update comes in.

There is no obsolete folder for the custom parts, so seems to be no way to mark a part version as out-dated. In addition, that reference to the new version would need to be added to the old part file. Nothing in the process of loading an fzpz file has the ability to modify an older version, even if there was a way to provided the upgrade path linkage.

It is quite possible to turn the custom parts folder structure into a git (or other) repository, and do manual pulls to update from somewhere else. That would not provide any part update options for existing instances of the older version. Fritzing keeps a copy of any used custom in the sketch, so updates like that should not affect any old parts in existing sketches.

To get where I think you intended, would need some rethinking, design, development of how custom parts are used, stored, updated in Fritzing. Starting with documenting what the user would expect to happen in various scenarios. The sketch MUST keep a copy of all custom parts, so that the document can be opened by another user, without needing to first get copies of and load all of the custom parts.

If I can offer some suggestion, I would have a look at artifact repository, and client tools. In addition to code versioning (svn github, …), it “seals” files into an artifact with a version. Can be integrated with CI server (which could build in our case the fzpz file giving the version number, releasing final version, checking rules, …)

I know quite well Maven (that for Java librairies). It has a very clear, accurate, strict … versioning process, efficient for development (SNAPSHOT), automatic update to last version in a range (or not). Client supports multiple artifact repositories, …

I use it also for other purposes than java libs and I’m quite satisfied of that choice. Maybe there are other similar advanced tool that can fit better with cpp…

The fzpz part files contain the same internal pieces (fzp definition and svg view images) as the core parts, but only the core parts are stored in, managed by, the core parts library. This is intended.

fzpz files are individual custom parts which can be created by anyone. They are not managed in any way by the code or part repositories. They are not managed by the application, until they are opened by a user, then they become available in the “MINE” bin. They are also maintained as part of any sketch (document) they are placed in.

A central (or list of) repository of those custom parts would be convenient, to be able to find what others have published. But do not want to add any extra ‘requirements’ to creating new parts. People can create parts for their own use, without ever publishing them.

Given the extremely distributed nature of custom parts creation, fixing, enhancing, with no ‘authority’ governing what is better, newer, derived from other parts, automating any sort of library with updates is not (from my view) practical. An update that fixes specific problems or limitations for a part can easily break something the original creator (or other users since) require. I see no way to differentiate between bug fixes, upgrades, and forks.

Most of the existing fzpz files I have looked at wold fail any sort of comprehensive checking rules. The same is true of the existing parts in the core library. But they are (somewhat unfortunately) good enough for many people to use. Tools are being worked on to try to raise the bar on what is ‘good enough’.

If you want to create a storage and retrieval mechanism, where fzpz parts can be published, queried, downloaded by a regular user, go for it. I wish you lots of luck. If you want to integrate that with the Fritzing app, so the parts can be found by a search and/or loaded by an extension to the ‘open’ command, that would need to be a suggestion (issue) to the application repository. With everything else on the list, that would be very low priority. Unless you found someone that wanted to do the work, and the update PRs get approved for inclusion.

While I’m not clear on exactly what you want, I think the mine parts bin may do some of it. Parts (.fzpz files) loaded in to the mine parts bin are a in github repository format internally to Fritzing (just like core parts, but in a different directory.) The main difference is if you use a part from the mine parts bin in a sketch, the part from your mine parts bin will be included in the .fzz file (because the part isn’t in core parts) and will thus appear in the temp parts bin extracted from the .fzz file an in the sketch on another Fritzing installation that doesn’t have your mine parts bin. There isn’t anything to stop you from creating a github repository of Fritzing parts (lots of people have) and putting your parts there. You can go as far as creating the necessary .fzb file to make a loadable bin with all of your parts that people can load as a single load (which will create a new section in the core parts window with your new parts.)

Peter