Magento have front-end and admin part of file upload option in themes so You can enable file upload by using custom option if you add file upload button.Frontend html is found in:
app/design/frontend/default/default/template/catalog/product/view/options/type/file.phtml
Frontend javascript code in:
app/design/frontend/default/default/template/catalog/product/view/options.phtml
(optionFileUpload js object = asynchronous iframe upload)
You can enable file upload custom option if you add:
<file translate="label" module="adminhtml">
<label>File</label>
<render>adminhtml/catalog_product_edit_tab_options_type_file</render>
<types>
<file translate="label" module="adminhtml">
<label>File Upload</label>
</file>
</types>
</file>
inside node:
config->global->catalog->product->options->custom->groups
in:
app/code/core/Mage/Catalog/etc/config.xml
Now you have new File Upload option that you can add to product!