Difference between revisions of "Fileplanet/uploadingfilestoia"
Megalanya0 (talk | contribs) m (MOTHERFUCKER ! ! !) |
m (Reverted edits by Megalanya0 (talk) to last revision by Schbirid) |
||
Line 1: | Line 1: | ||
Brainstorming on how we can upload the files to IA items. | |||
'''Remember not to upload the ftp2 stuff publically!''' | |||
'''x-archive-meta-title:''' "Fileplanet_Path_with_underscores". Without the ftp1/2/3 bit, so eg "ftp1/102011/Yes_Man_Dynamic_Theme.7z" would become the item "102011_Yes_Man_Dynamic_Theme.7z", "ftp1/fpnew/patches/prorally2001_v11.exe" would be "fpnew_patches_prorally2001_v11.exe". This might need some care about special characters that IA does not support for item names. | |||
'''x-archive-meta-description:''' For now just put the path there. So basically the item name but with special characters intact. TODO later is to add the real metadata we got, shaqfu has a sqlite db with it, otherwise use the fileinfo item: http://archive.org/details/FileplanetFiles_fileinfo_pages_images | |||
'''x-archive-meta-date:''' Not sure what format it supports, but this should be the file's original timestamp. | |||
'''x-archive-meta-year:''' See above. | |||
'''x-archive-meta-subject:''' gaming;software;gaming software;fileplanet;gamespy;ign;planetnetwork | |||
'''x-archive-meta-collection:'''archiveteam-fileplanet no idea if this would work with s3 upload | |||
'''x-archive-meta-mediatype:'''software (not always correct, maybe check for filename extensions like avi, mov, mp4 etc to set it to movies for those) | |||
'''Remember not to upload the ftp2 stuff publically!''' | |||
#!/bin/bash | |||
# traverse through subdirectories, generate metadata | |||
# supply eg ftp1/102009/ as argument, it will then upload that directory | |||
parentdirectory=$(dirname $1) # NO trailing slash! | |||
subdirectory=$(basename $1) | |||
echo "is ${parentdirectory}/${subdirectory} the directory you want to upload?" | |||
read | |||
commonheaders='--add-header x-archive-auto-make-bucket:1 --add-header x-archive-meta-noindex:true --add-header "x-archive-meta-subject: gaming;software;fileplanet;gamespy;ign;planetnetwork" --add-header "x-archive-meta-collection:archiveteam-fileplanet" --add-header "x-archive-meta-mediatype:software"' | |||
# mediatype:software is not always correct, maybe check for filename extensions like avi, mov, mp4 etc to set it to movies for those?) | |||
## nah, underscor said software :) | |||
tempfile="/tmp/fileplanet_ListOfFiles" | |||
echo "Generating a list of files to upload" | |||
find ${parentdirectory}/${subdirectory} -type f > ${tempfile} | |||
while read file | |||
do | |||
file=$(echo ${file}| sed "s/\.\///") # remove ./ | |||
echo "Now uploading ${file}" | |||
datetime=$(ls -l --time-style=long-iso "${file}" | awk '{print $6" "$7}') | |||
year=$(echo ${datetime} | grep -Eo '[0-9]{4}') | |||
date="--add-header x-archive-meta-date:\"${datetime}\"" | |||
year="--add-header x-archive-meta-year:${year}" | |||
filename=$(basename "${file}") | |||
title="--add-header x-archive-meta-title:\"Fileplanet Archive: ${filename}\"" | |||
desc="--add-header x-archive-meta-description:\"${filename}, mirrored from its original location in ${file}\"" | |||
# from famicoman | |||
# IA supports alphanum and _-. | |||
itemname=$(echo "Fileplanet_${file}" | tr ' ' '_' | tr -d '[{}(),\!:?~@#$%^&*+=;<>|]' | tr -d "\'" | sed 's/\//_/g') | |||
echo "s3cmd ${commonheaders} ${date} ${year} ${title} ${desc} put \"${file}\" s3://${itemname}" | |||
echo "#################" | |||
done < ${tempfile} | |||
rm ${tempfile} |
Revision as of 16:24, 16 January 2017
Brainstorming on how we can upload the files to IA items.
Remember not to upload the ftp2 stuff publically!
x-archive-meta-title: "Fileplanet_Path_with_underscores". Without the ftp1/2/3 bit, so eg "ftp1/102011/Yes_Man_Dynamic_Theme.7z" would become the item "102011_Yes_Man_Dynamic_Theme.7z", "ftp1/fpnew/patches/prorally2001_v11.exe" would be "fpnew_patches_prorally2001_v11.exe". This might need some care about special characters that IA does not support for item names.
x-archive-meta-description: For now just put the path there. So basically the item name but with special characters intact. TODO later is to add the real metadata we got, shaqfu has a sqlite db with it, otherwise use the fileinfo item: http://archive.org/details/FileplanetFiles_fileinfo_pages_images
x-archive-meta-date: Not sure what format it supports, but this should be the file's original timestamp.
x-archive-meta-year: See above.
x-archive-meta-subject: gaming;software;gaming software;fileplanet;gamespy;ign;planetnetwork
x-archive-meta-collection:archiveteam-fileplanet no idea if this would work with s3 upload
x-archive-meta-mediatype:software (not always correct, maybe check for filename extensions like avi, mov, mp4 etc to set it to movies for those)
Remember not to upload the ftp2 stuff publically!
#!/bin/bash # traverse through subdirectories, generate metadata # supply eg ftp1/102009/ as argument, it will then upload that directory parentdirectory=$(dirname $1) # NO trailing slash! subdirectory=$(basename $1) echo "is ${parentdirectory}/${subdirectory} the directory you want to upload?" read commonheaders='--add-header x-archive-auto-make-bucket:1 --add-header x-archive-meta-noindex:true --add-header "x-archive-meta-subject: gaming;software;fileplanet;gamespy;ign;planetnetwork" --add-header "x-archive-meta-collection:archiveteam-fileplanet" --add-header "x-archive-meta-mediatype:software"' # mediatype:software is not always correct, maybe check for filename extensions like avi, mov, mp4 etc to set it to movies for those?) ## nah, underscor said software :) tempfile="/tmp/fileplanet_ListOfFiles" echo "Generating a list of files to upload" find ${parentdirectory}/${subdirectory} -type f > ${tempfile} while read file do file=$(echo ${file}| sed "s/\.\///") # remove ./ echo "Now uploading ${file}" datetime=$(ls -l --time-style=long-iso "${file}" | awk '{print $6" "$7}') year=$(echo ${datetime} | grep -Eo '[0-9]{4}') date="--add-header x-archive-meta-date:\"${datetime}\"" year="--add-header x-archive-meta-year:${year}" filename=$(basename "${file}") title="--add-header x-archive-meta-title:\"Fileplanet Archive: ${filename}\"" desc="--add-header x-archive-meta-description:\"${filename}, mirrored from its original location in ${file}\"" # from famicoman # IA supports alphanum and _-. itemname=$(echo "Fileplanet_${file}" | tr ' ' '_' | tr -d '[{}(),\!:?~@#$%^&*+=;<>|]' | tr -d "\'" | sed 's/\//_/g') echo "s3cmd ${commonheaders} ${date} ${year} ${title} ${desc} put \"${file}\" s3://${itemname}" echo "#################" done < ${tempfile} rm ${tempfile}