# AEM Asset Upload Sample Application

This sample application demonstrates programmatic asset upload to **AEM as a Cloud Service** using the [`@adobe/aem-upload`](https://github.com/adobe/aem-upload) Node.js library.

## ⚡ Quick Start

Get up and running in 5 minutes:

```bash
# 1. Install dependencies
npm install

# 2. Configure environment
cp env.example .env
# Edit .env with your AEM credentials

# 3. Verify setup (optional but recommended)
npm run verify

# 4. Run the application
npm start
```

Choose **Option 1 (FileSystemUpload)** to upload the sample files and see it in action!

## Overview

The application showcases three different approaches to uploading assets:

1. **FileSystemUpload** - Upload files from local file system with directory structure support and auto-folder creation
2. **DirectBinaryUpload** - Fine-grained control over individual file uploads from URLs, streams, or buffers
3. **Batch Upload** - Upload multiple files in batches with automatic retry logic and error recovery

All examples leverage the **direct binary upload** approach, which offers significant performance improvements by uploading directly to cloud storage (Azure Blob Storage), bypassing AEM's Java processes.

## Prerequisites

Before running this application, ensure you have:

- ✅ **Node.js** (version 18 or higher) - [Download here](https://nodejs.org/)
- ✅ **AEM as a Cloud Service** author environment (RDE, Dev, Stage, or Prod)
- ✅ **Valid credentials** for your AEM environment (username/password or bearer token)

> **⚠️ Important:** You **cannot** use the AEM SDK for local development. You must have access to an actual AEM as a Cloud Service environment.

## Installation

### Step 1: Install Dependencies

Install the required Node.js packages:

```bash
npm install
```

This will install:

- `@adobe/aem-upload` - The core upload library
- `@adobe/jwt-auth` - JWT authentication for service credentials
- `dotenv` - For environment variable management
- `chalk` - For colored console output
- `ora` - For progress spinners

### Step 2: Configure Environment Variables

Create a `.env` file with your AEM credentials:

```bash
cp env.example .env
```

Edit the `.env` file and provide your AEM details:

```properties
# AEM as a Cloud Service Author URL (without trailing slash)
AEM_URL=https://author-p12345-e67890.adobeaemcloud.com

# Authentication (choose one method)
# Method 1: Service Credentials (RECOMMENDED for production)
AEM_SERVICE_CREDENTIALS_FILE=./service-credentials.json

# Method 2: Bearer Token (for manual testing)
# AEM_BEARER_TOKEN=your-bearer-token

# Method 3: Basic Authentication (for development/testing only)
# AEM_USERNAME=your-username
# AEM_PASSWORD=your-password

# Target folder in AEM DAM
TARGET_FOLDER=/content/dam

# For DirectBinaryUpload example - Remote file URL
REMOTE_FILE_URL_1=https://placehold.co/600x400/red/white?text=Adobe+Experience+Manager+Assets

# Optional: Enable debug logging
DEBUG=false
```

> **💡 Tip:** For production use, Service Credentials authentication is **strongly recommended** as it provides automatic token refresh and enhanced security.

### Step 3: Verify Setup (Recommended)

Before uploading, verify your configuration:

```bash
npm run verify
```

This will check:

- ✅ Node.js version
- ✅ Installed dependencies
- ✅ Environment configuration
- ✅ Sample assets availability
- ✅ AEM connectivity

## Authentication Methods

This application supports three authentication methods. Choose the one that best fits your use case:

### Method 1: Service Credentials (RECOMMENDED for Production) 🔐

Service Credentials provide the most secure and production-ready authentication method with automatic token refresh.

#### What are Service Credentials?

Service Credentials are JSON files downloaded from the AEM Developer Console that contain:

- Client ID and Client Secret
- Technical Account ID
- Organization ID
- Private Key for JWT signing
- Meta Scopes (permissions)

#### How to Obtain Service Credentials

Follow these steps to download service credentials from AEM as a Cloud Service:

**Step 1: Access AEM Developer Console**

1. Log in to [Adobe Cloud Manager](https://experience.adobe.com/#/@aem/cloud-manager)
2. Open your **Program** containing the AEM environment you want to access
3. In the **Environments** section, click the **"..."** (ellipsis) next to your environment
4. Select **Developer Console**

**Step 2: Create or Access Technical Account**

1. In the Developer Console, click the **Integrations** tab
2. Click the **Technical Accounts** tab
3. Either:
   - **Existing Account:** Expand an existing Technical Account
   - **New Account:** Click **Create new technical account** button

**Step 3: Download Service Credentials**

1. Expand the **Private Key** section (it should show status: **Active**)
2. Click the **"..." > View** button associated with the Private Key
3. The Service Credentials JSON will be displayed
4. Click the **download button** in the top-left corner
5. Save the file as `service-credentials.json` in your project root

**Step 4: Configure Your Application**

1. Save the downloaded JSON file in your project directory (e.g., `service-credentials.json`)
2. Update your `.env` file:

   ```properties
   AEM_SERVICE_CREDENTIALS_FILE=./service-credentials.json
   ```

3. **IMPORTANT:** The file is automatically added to `.gitignore` - **NEVER commit service credentials to version control!**

**Step 5: Grant Permissions in AEM**

The technical account needs proper permissions in AEM:

1. Log in to your AEM Author environment as an administrator
2. Navigate to **Tools** > **Security** > **Users**
3. Find the technical account user (login name from `integration.email` in your JSON)
4. Open the user's **Properties**
5. Go to the **Groups** tab
6. Add the user to the **DAM Users** group (for write access to assets)
7. Click **Save and Close**

#### Benefits of Service Credentials

- ✅ **Automatic Token Refresh** - No manual token generation needed
- ✅ **Long-lived** - Service credentials are valid for 365 days
- ✅ **Secure** - Private key never exposed in requests, only used for JWT signing
- ✅ **Production-ready** - Recommended by Adobe for all automated workflows
- ✅ **Token Caching** - Access tokens are cached and reused until expiration

#### Example Service Credentials Structure

See `service-credentials.example.json` for the expected JSON structure.

### Method 2: Bearer Token (For Manual Testing) 🔑

Use a pre-generated bearer token for quick testing and development.

**How to Get a Bearer Token:**

1. Log in to AEM Developer Console (same steps as above)
2. Navigate to **Integrations** > **Local Development Token**
3. Click **Get Local Development Token**
4. Copy the token
5. Add it to your `.env` file:

   ```properties
   AEM_BEARER_TOKEN=your-token-here
   ```

**Limitations:**

- ⚠️ Expires after 24 hours
- ⚠️ Must be manually regenerated
- ⚠️ Not suitable for automated workflows

### Method 3: Basic Authentication (Development Only) 🔓

Use your AEM username and password for local development and testing.

**Configuration:**

```properties
AEM_USERNAME=your-username
AEM_PASSWORD=your-password
```

**Limitations:**

- ⚠️ Less secure than token-based authentication
- ⚠️ Not recommended for production
- ⚠️ Should only be used in development environments
- ⚠️ Requires a valid AEM user account


### Which Authentication Method Should I Use?

| Use Case | Recommended Method |
|----------|-------------------|
| **Production deployments** | Service Credentials (Method 1) |
| **CI/CD pipelines** | Service Credentials (Method 1) |
| **Automated workflows** | Service Credentials (Method 1) |
| **Quick testing/debugging** | Bearer Token (Method 2) |
| **Local development** | Basic Auth (Method 3) or Bearer Token (Method 2) |

## Usage

### Interactive Mode

Run the main application to access an interactive menu:

```bash
npm start
```

You'll see a menu where you can select which example to run:

```
╔════════════════════════════════════════════════════════════╗
║      AEM Asset Upload Sample Application                  ║
║      Demonstrating @adobe/aem-upload library               ║
╚════════════════════════════════════════════════════════════╝

Choose an upload method:

1. FileSystemUpload - Upload files from local file system
2. DirectBinaryUpload - Fine-grained control over uploads
3. Batch Upload - Upload multiple files with retry logic
4. Exit
```

### Running Individual Examples

You can also run each example directly:

#### 1. FileSystemUpload Example

```bash
npm run filesystem-upload
```

This example:

- Uploads all files from the `sample-assets` directory
- Automatically creates folders in AEM if they don't exist
- Maintains directory structure
- Ideal for bulk uploads and migrations

**Best for:** Uploading local files while preserving folder structures.

#### 2. DirectBinaryUpload Example

```bash
npm run direct-binary-upload
```

This example:

- Uploads files from remote URLs (configured in `.env`)
- Provides fine-grained control over each file
- Suitable for uploads from streams, buffers, or remote sources
- Requires target folders to already exist in AEM

**Best for:** Uploading from remote sources, APIs, or processing files before upload.

#### 3. Batch Upload Example

```bash
npm run batch-upload
```

This example:

- Uploads files in configurable batches (demo: 2 items per batch)
- Includes automatic retry logic (up to 3 attempts with exponential backoff)
- Shows performance metrics and upload speed
- Production-ready error handling

**Best for:** Production environments, large-scale uploads, unreliable networks.

## Sample Assets

The `sample-assets` directory contains sample files for testing:

```
sample-assets/
├── Freshness.png              # Sample image
├── doc/
│   └── aem-cloud-service-security-overview.pdf
├── imgs/
│   ├── 600x600.png
│   └── 1600x900.png
└── video/
    └── where-are-we.mp4
```

You can:

1. Use the provided sample files
2. Add your own files to this directory
3. Replace with files you want to upload

**Supported file types:** All file types supported by AEM Assets (images, PDFs, videos, documents, etc.)

## Verifying Your Uploads

After a successful upload, you'll see a clickable URL in the terminal output:

```
✅ Successfully uploaded to AEM: https://author-p12345-e67890.adobeaemcloud.com/ui#/aem/assets.html/content/dam?appId=aemshell
  → Freshness.png
  → doc/aem-cloud-service-security-overview.pdf
  → imgs/600x600.png
  → imgs/1600x900.png
  → video/where-are-we.mp4
```

**Click the URL** to open the AEM Assets UI and view your uploaded files directly in your browser!

## How It Works

### Direct Binary Upload Flow

```
┌─────────────────┐
│ Client App      │
│ (This Sample)   │
└────────┬────────┘
         │ 1. Initiate Upload
         ▼
┌─────────────────┐
│ AEM as a Cloud  │
│ Service         │
└────────┬────────┘
         │ 2. Return Presigned URLs
         ▼
┌─────────────────┐
│ Client App      │
└────────┬────────┘
         │ 3. Upload Binary to Cloud
         ▼
┌─────────────────┐
│ Cloud Storage   │
│ (S3/Azure Blob) │
└─────────────────┘

┌─────────────────┐
│ Client App      │
└────────┬────────┘
         │ 4. Notify Completion
         ▼
┌─────────────────┐
│ AEM as a Cloud  │
│ Service         │
└────────┬────────┘
         │ 5. Trigger Processing
         ▼
┌─────────────────┐
│ Adobe Asset     │
│ Processing      │
└─────────────────┘
```

### Key Benefits

- **🚀 Faster Uploads** - Direct to cloud storage, bypassing AEM Java processes
- **⚡ Reduced Server Load** - AEM only handles metadata and orchestration
- **📦 Batch Support** - Upload multiple files efficiently
- **🔄 Automatic Retries** - Built-in error handling
- **📊 Progress Tracking** - Real-time upload progress

## Code Structure

```
aem-asset-upload-sample/
├── index.js                            # Main interactive CLI
├── verify-setup.js                     # Setup verification script
├── package.json                        # Dependencies and scripts
├── env.example                         # Environment configuration template
├── service-credentials.example.json    # Service credentials template
├── README.md                           # This file
├── SERVICE-CREDENTIALS-SETUP.md        # Quick setup guide for service credentials
├── .gitignore                          # Git ignore patterns
├── src/
│   ├── config.js                       # Configuration management
│   ├── auth.js                         # Service credentials & JWT authentication
│   ├── errorHandler.js                 # Centralized error handling and reporting
│   └── utils.js                        # Utility functions (progress, logging, URL construction)
├── examples/
│   ├── filesystem-upload.js            # FileSystemUpload example (auto-creates folders)
│   ├── direct-binary-upload.js         # DirectBinaryUpload example (remote URLs)
│   └── batch-upload.js                 # Batch upload with retry logic
└── sample-assets/                      # Sample files for testing
    ├── Freshness.png
    ├── doc/
    │   └── aem-cloud-service-security-overview.pdf
    ├── imgs/
    │   ├── 600x600.png
    │   └── 1600x900.png
    └── video/
        └── where-are-we.mp4
```

## Understanding the Examples

### FileSystemUpload Class

**When to use:**

- Uploading files from a local directory
- Maintaining directory structures
- Simple bulk uploads

**Example usage:**

```javascript
const { FileSystemUpload } = require('@adobe/aem-upload');
const upload = new FileSystemUpload();
const uploadResult = await upload.upload(options, [localDirectory]);
```

### DirectBinaryUpload Class

**When to use:**

- Uploading from streams or buffers
- Custom file handling logic
- More control over individual file uploads
- Uploading generated or processed content

**Example usage:**

```javascript
const { DirectBinaryUpload } = require('@adobe/aem-upload');
const upload = new DirectBinaryUpload();
const uploadResult = await upload.uploadFiles(options, uploadFiles);
```

## Troubleshooting

### Common Issues

#### 1. "AEM_URL is required"

**Problem:** The `.env` file is missing or not configured.

**Solution:** 

- Copy `env.example` to `.env`
- Fill in your AEM as a Cloud Service URL and credentials

#### 2. "Authentication failed" or "Service credentials file not found"

**Problem:** Invalid credentials or missing service credentials file.

**Solution:**

- **For Service Credentials:**
  - Verify the file path in `AEM_SERVICE_CREDENTIALS_FILE` is correct
  - Ensure the JSON file exists and contains valid service credentials
  - Check that the file was downloaded from AEM Developer Console
  - Verify the service credentials haven't expired (365-day validity)
- **For Bearer Token:**
  - Verify your bearer token is still valid (expires after 24 hours)
  - Regenerate a new token from AEM Developer Console
- **For Basic Auth:**
  - Verify your username/password are correct
  - Ensure your user account is active in AEM
- **For All Methods:**
  - Ensure the technical account/user has write permissions to the target folder
  - Check that the AEM URL is correct (no trailing slash)

#### 3. "Failed to generate access token from service credentials"

**Problem:** Unable to generate JWT or exchange for access token.

**Solution:**

- Verify your service credentials haven't expired (check the creation date)
- Ensure you have network connectivity to Adobe IMS (`ims-na1.adobelogin.com`)
- Check that the private key in the JSON is complete and properly formatted
- Verify the client secret hasn't been rotated
- Generate new service credentials from AEM Developer Console if needed

#### 4. "403 Forbidden" when uploading assets

**Problem:** The technical account doesn't have proper permissions in AEM.

**Solution:**

- Log in to AEM as an administrator
- Navigate to **Tools** > **Security** > **Users**
- Find the technical account user (from `integration.email` in your service credentials)
- Add the user to the **DAM Users** group or create a custom group with appropriate permissions
- Verify the user has write access to the target folder

#### 5. "Cannot use AEM SDK"

**Problem:** Trying to use with local AEM SDK.

**Solution:**

- Direct binary upload is **only available** on AEM as a Cloud Service
- Use an RDE, Dev, or other cloud environment

#### 6. "No files found in sample-assets"

**Problem:** The sample-assets directory is empty.

**Solution:**

- Add files to the `sample-assets` directory
- Or modify the examples to point to your own directory

### Enable Debug Logging

Set `DEBUG=true` in your `.env` file to see detailed logs:

```properties
DEBUG=true
```

## Best Practices

### For Development

1. **Start with FileSystemUpload** - It's the easiest to understand and automatically creates folders
2. **Use Rapid Development Environment (RDE)** - Perfect for testing without affecting other environments
3. **Enable debug logging** - Set `DEBUG=true` in `.env` to see detailed logs
4. **Test with small files first** - Verify your setup before uploading large files
5. **Run the verify script** - Use `npm run verify` to catch configuration issues early

### For Production

1. **Use Service Credentials Authentication** - Provides automatic token refresh and enhanced security
2. **Secure Credential Storage** - Store service credentials files securely, never commit to version control
3. **Implement Retry Logic** - Network issues are common; use the batch upload pattern
4. **Configure Appropriate Batch Sizes** - Based on file sizes and network speed (typically 10-50 items)
5. **Monitor Upload Progress** - Track large uploads and provide feedback to users
6. **Validate File Types** - Ensure files are AEM-compatible before uploading
7. **Set Appropriate Timeouts** - Large files need longer timeouts
8. **Handle Errors Gracefully** - Provide clear error messages and recovery options
9. **Check User Permissions** - Ensure the technical account has write access to target folders
10. **Rotate Service Credentials Regularly** - Generate new credentials before the 365-day expiration

## Extending This Sample

This sample is designed to be a starting point for your own implementation. Here are some ideas:

### Customization Ideas

- **Metadata Management** - Add custom metadata during upload
- **Folder Organization** - Implement custom folder structures based on file types or dates
- **CSV Upload** - Read file paths from CSV and upload in bulk
- **Watch Folder** - Monitor a folder and auto-upload new files
- **Integration** - Connect with other systems (CMS, DAM migration, etc.)


## Additional Resources

- [aem-upload Library Documentation](https://github.com/adobe/aem-upload)
- [AEM as a Cloud Service Documentation](https://experienceleague.adobe.com/docs/experience-manager-cloud-service.html)
- [Service Credentials Authentication Guide](https://experienceleague.adobe.com/en/docs/experience-manager-learn/getting-started-with-aem-headless/authentication/service-credentials) - Official Adobe tutorial
- [Direct Binary Upload API](https://experienceleague.adobe.com/docs/experience-manager-cloud-service/content/assets/admin/developer-reference-material-apis.html)
- [AEM Assets HTTP API](https://experienceleague.adobe.com/docs/experience-manager-cloud-service/content/assets/admin/mac-api-assets.html)
- [@adobe/jwt-auth npm package](https://www.npmjs.com/package/@adobe/jwt-auth)

## License

This sample application is provided under the MIT License.

## Support

For issues with:

- **This sample application** - Check the code and configuration
- **AEM as a Cloud Service** - Contact Adobe Support
- **@adobe/aem-upload library** - Submit issues on [GitHub](https://github.com/adobe/aem-upload/issues)

**Happy Uploading! 🚀**

