How to import Stations Open Data provided by the French governement

The French government, using its open data portal https://www.data.gouv.fr, is providing a monthly updated list of stations. The dedicated page is here:

And the last update is here:

https://www.data.gouv.fr/fr/datasets/r/50625621-18bd-43cb-8fde-6b8c24bdabb3

Do you have a standard file import format? Maybe it would be possible to transform this file from the French government in a data format that suits you?

1 Like

Thanks, it looks like this also has an API so we can find the latest version. I’ll investigate to see if I can build an import for this as it has an open data license.

So based on the first draft of this import, this data has 10282 items, of which 5986 will be skipped as duplicates (either in the data or because we appear to have them already), 4296 POIs will be added.

Hello Christopher,
I’ve requested recently to add the operator Sigief and become to implement by hand the some CCS station around me.

I’ve realized that the information are already in the file from the source you already imported it.

Could you import the latest version of this file, all the SIGEIF is inside as per today.
https://www.data.gouv.fr/fr/datasets/r/8d9398ae-3037-48b2-be19-412c24561fbb
last update : 13 mai 2022

Thanks

I’ll look into what’s required, some imports are difficult to repeat if we have keyed on an ID that later changes.

From my understanding you have already imported a general set of french data organized IRVE, so if dataset are structure in the same way, it should be straightforward.
The hint should be in the tag (metadonnées) “irve” :

If you confirm me that IRVE structure is ok, and block on some french data structured not according to IRVE, let me know, I’ll convert them for you. (Especially the other set that I’ve found)
Likely you’ll not be able to have a script to pull them monthly, but neither the stations will grow significantly by months :slight_smile:
Once you use the dataset let me know I’ll define openchargemap as a data user on the dataset, closing the opensource loop :slight_smile:

So digging more in the data set, there is one main dataset irve, that you already imported in 03/20, and multiple small one that are not included… like the Sigief, Tesla, ionity etc…
If you confirm that the IRVE datable format is still ok, I can concatenate all the dataset relevant from this website and provide you one file.

what we need is a single API call to exort all the data, previoulsy we used this https://www.data.gouv.fr/api/1/datasets/?q=5448d3e0c751df01f85d0572&page=0&page_size=20 which doesn’t work anymore, so we’d need to track down the right API call. The export must include all network operators, not one at a time.

I had actually given up on most imports like this because the data is fairly mixed quality and in some parts is wrongly aggregated. Instead we want networks to agree to provide us data in a standard format directly: https://openchargemap.org/site/about/datasharing - currently only one network has offered to do that and is yet to provide us the necessary API credentials to work on that.

I appreciate your offer of concatenating data etc but that’s not a repeatable/sustainable way of working.

Thanks for your feedback,

I agree with you on all the points, however I would like to add :

  • In my area, significant amount of stations are missing (mainly the one that are not included in the file that you importer 2 years ago). A one time import, like you already did would help significantly to address these missing networks.
  • This french server allow to use an API to download files (doc in french :Documentation de API catalogue des donnĂ©es ouvertes - data.gouv.fr - api.gouv.fr)
    As you mentioned each dataset version has a unique ID, that make your old link obsolete. Not the best for recurrent data pulling. However I’m confident, that with a script (curl based), the latest dataset version ID could be retrieve, thus the data linked to this ID. (It’s about procession json payload)
    My capabilities are only in python/bash, I could try to generate a script pulling all the relevant french data and concatening them in one relevant dataframe.

So through my previous messy msg there is 2 different objectives :

  • One time import, wider than the previous one, to compensate most of the missing network in France.
  • Regular update, monthly or 6 month base to keep the database update.

These data are quite valuable, generated by the network owner shared in a opensource database, I’m not aware of any country centralizing such data in one place (however several files).

Even a one shot import, as you did 2 years ago, is really interesting as new chargers are not build every morning :wink:

Ps: Could you clarify what does mean " is wrongly aggregated." ?
Non maching dataframe structure ?

Thanks, here is the code for our existing import from this provider, if we can find the latest dataset ID then I can try that: ocm-system/ImportProvider_DataGouvFr.cs at master · openchargemap/ocm-system · GitHub

Other countries (in particular the UK) do have other open data sets but they all suffer from inconsistent storage of connector, equipment information, badly formatted addresses and incorrect latitude/longitude information (sometime swapped so points end up in the sea etc). While it’s technically possible to do more to refine and work around these problems it’s not something I’m going to spend time on, someone else is welcome to try though.

Thanks Christopher,

I’ve started to fork a python script from the french gov to generate a consolidate file, I’ll tune it for our need and data cleanup.
Will see what I’ll get.

Hello

Data has been updated on 17 July 2023 and contains about 43200 inputs.

Hello Frederic,

Are you mentioning the update from data gouv.fr ?
The file IRVE is a synchro of whoever submit his charging station database within a specific format.

FIY, I’ve submitted a processed file, a year ago, to christopher, after processing it he was able to add 4007 locations and updated 1315 others :love_you_gesture:
The output looked like (The color was the different network):

I could run again my script to pre-process the IRVE dataset for Christopher. Remind me that in a month or so :slight_smile:

Hi, sorry I don’t think we can import from that source again. It’s just not good enough quality and doesn’t have consistent information across versions (we need persistent ID references etc).

What we really need are OCPI data feeds directly from the networks with permission for us to redistribute the data as Open Data: https://openchargemap.org/site/about/datasharing

Right, that’s directly provided by the owner submitting the dataset on data.gouv

I quite remember some variation on the ID. Can you tell us more about that ID. How should it be formated and how is it used ?

Following yoyr answer i could eventually add a criteria to submit you only the new stations with clean/correct ID…even if we add “only” 50 stations, that’s already a win ?

Do you have a lot of french (or not) company using OCPI data feed ?

What would be the approach :

  • Writing a standardized email and shooting it to the mail box of the network owner ?
  • What about, if they have questions ? We forward you the frenchi msg ?

Cheer

Regarding IDs, within a given data set being imported we expect to have a unique ID reference per item which stays constant. So if we look at the data again in 6 months the same item should have the same ID. If that changed then a charging location that was in Paris might be replaced by one in Lille which now uses that ID. Or, if the ID is unique but has completely changed we will only se that we already have something at that approximate location and we will therefore have to ignore that item.

A list of new items we definitely don’t have would be great, but I’d imagine that’s a lot of work to create.

We have only a few OCPI feeds currently and none are for France, we would like to add more of these in preference to any other import work.

Yes, emailing each network and telling them about Open Charge Map would be ideal, in general we need to:

  • Introduce Open Charge Map and explain you are one of the volunteers helping the project.
  • Explain that it’s used by hundreds of apps and services globally as a source for EV charging location information and serves millions of queries per month.
  • Explain networks get free global and local advertising for their network, resulting in more users for their network. That some of their locations may already be listed and this is an opportunity for them to keep the information accurate.
  • Invite them to complete the data sharing agreement at: https://openchargemap.org/site/about/datasharing
  • Direct them to https://community.openchargemap.org/ if they have questions

Our data sharing page has the following intro paragraph, which says much the same:

We invite all charging networks to promote their charging locations via Open Charge Map by directly supplying EVSE charging station location information to us for distribution. The benefits to your network of sharing your data with us include:

  • Promoting your network via our API and shared dataset to thousands of apps and services, delivering information to millions of EV drivers.
  • Ensuring we can distribute the latest correct information.
  • Our project goal is to distributing accurate information as widely as possible. We are not commercially related to any charging networks.

I’ve also removed the text in our data sharing agreement suggestion their would be a fee for imports, as that idea clearly isn’t going to work.

The script should still be running :), thus getting an updated dataset based. From that, I could just run a diff between this one and the legacy one I provided, to give a process one with only the new entries.
Then you could add/update as last year, I should have some time end of august.

What would be interesting is to generate a template that the community could translate via online contribution tool (I’ve done it several time for github project, but I cannot remember the tool name)
also applying the same to the page /about/datasharing etc…

  • Thus you’ll somewhat ensure the message carried by the community to the network owner correctly shaped.
  • Offering template in native language to the community, we could mass fire the msg (no spam :slight_smile: ) to the network owner.
  • Simply offering native language website would allow a wider use.

I’ve though couple of time to send a msg to network owner, but I’ve been lazy to build a formal and clean email regarding OCM + sharing to them an english technical webpage … hum, I’ll have max 1 over 10 chance to have the recipient understanding the title … :slight_smile:

1 Like

That language issue is difficult, we did previously have translated elements on our site but the major issue was that translators only made a one time effort to supply the text, and if we made changes to english content we didn’t have a way to get all translations updated. We still have this issue with out app but we removed as much text as possible from it.

I’d prefer an automated option which gives users an auto translated link, one which you and any other people could review first before sending out.

So my point is, let’s not get fixated on translating the text of the website into other languages right now, we’d first have to prove hundreds of operators are open to the idea of sharing their data (which so far, they’ve not been).

I’m sure everyone competent enough (and with authority enough) to complete the form knows you can translate any webpage using your own browser - that’s exactly what I do when visiting non-english sites, it’s not a big problem.