Skip to content
GitLab
Menu
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
Observatoire
observatoire-scripts
Commits
b17a431e
Commit
b17a431e
authored
Sep 30, 2019
by
Pierre Dittgen
Browse files
Fixes
parent
7461c71c
Changes
2
Hide whitespace changes
Inline
Side-by-side
README.md
View file @
b17a431e
...
...
@@ -18,12 +18,12 @@
## Dépendances techniques
-
wget
-
mkdir
-
bash
-
csvcut (from
[
csvkit
](
https://csvkit.readthedocs.io/en/latest/
)
package)
-
sed
-
csvcut
, csvjoin, csvsort
(from
[
csvkit
](
https://csvkit.readthedocs.io/en/latest/
)
package)
-
grep
-
iconv
-
sed
-
wget
#-------------------------------------------
...
...
process_and_generate
View file @
b17a431e
...
...
@@ -19,12 +19,12 @@ generate_organizations_csv() {
mkdir
-p
$ORG_TEMP_DIR
$CSV_CUT
-c
"siren,type,url-ptf,url-datagouv,id-datagouv"
$ODF_ORGA_FILE
>
$ORG_TEMP_DIR
/odf_org_1.csv
$CSV_GREP
-c
"statut"
-m
"ok"
$CACHE_DIR
/siren_info.csv |
$CSV_CUT
-C
"statut,message"
>
$ORG_TEMP_DIR
/siren_ok_info.csv
$CSV_JOIN
-c
1
$ORG_TEMP_DIR
/odf_org.csv
$ORG_TEMP_DIR
/siren_ok_info.csv
>
$ORG_TEMP_DIR
/odf_org_2.csv
$CSV_JOIN
-c
1
$ORG_TEMP_DIR
/odf_org
_1
.csv
$ORG_TEMP_DIR
/siren_ok_info.csv
>
$ORG_TEMP_DIR
/odf_org_2.csv
$CSV_JOIN
-c
"code_departement,depcode"
$ORG_TEMP_DIR
/odf_org_2.csv
$CACHE_DIR
/cog_departement.csv
>
$ORG_TEMP_DIR
/odf_org_3.csv
$CSV_JOIN
-c
"code_region,regcode"
$ORG_TEMP_DIR
/odf_org_3.csv
$CACHE_DIR
/cog_region.csv
>
$ORG_TEMP_DIR
/odf_org_4.csv
ORGANIZATIONS_HEADER
=
"siren,nom,type,url-website,url-datagouv,id-datagouv,reg-code,reg-nom,dep-code,dep-nom,lat,long"
(
echo
$ORGANIZATIONS_HEADER
&&
$CSV_CUT
-c
"siren,nom,type,url-ptf,url-datagouv,id-datagouv,code_region,regnom,code_departement,depnom,latitude,longitude"
$ORG_TEMP_DIR
/odf_org_4.csv |
$SED
"1d"
)
>
$BUILD_DIR
/organizations.csv
(
echo
$ORGANIZATIONS_HEADER
&&
$CSV_CUT
-c
"siren,nom,type,url-ptf,url-datagouv,id-datagouv,code_region,regnom,code_departement,depnom,latitude,longitude"
$ORG_TEMP_DIR
/odf_org_4.csv |
$SED
"1d"
)
>
$
OBS_
BUILD_DIR
/organizations.csv
rm
-fR
$ORG_TEMP_DIR
}
...
...
@@ -33,8 +33,10 @@ generate_websites_csv() {
WS_TEMP_DIR
=
$CACHE_DIR
/website
mkdir
-p
$WS_TEMP_DIR
$CSV_CUT
-c
"nom,url,techno,porteur,contact,twitter"
$ODF_PTF_FILE
>
$BUILD_DIR
/websites.csv
$CSV_CUT
-c
"nom,url,techno,porteur,contact,twitter"
$ODF_PTF_FILE
>
$
OBS_
BUILD_DIR
/websites.csv
}
OBS_BUILD_DIR
=
$BUILD_DIR
/observatoire
mkdir
-p
$OBS_BUILD_DIR
generate_organizations_csv
generate_websites_csv
\ No newline at end of file
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment